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ABSTRACT 


| 

This  is  the  first  of  a series  of  papers  constructing  an  information 
based  general  theory  of  optimal  errors  and  analytic  computational  complexity. 
Among  the  applications  are  such  traditionally  diverse  areas  as  approximation, 
boundary- value  problems,  quadrature,  and  nonlinear  equations  in  a finite  or 
infinite  dimensional  space. 

Traditionally  algorithms  are  often  derived  by  ad  hoc  criteria.  The 
information  based  theory  rationalizes  the  synthesis  of  algorithms  bv  showing 
how  to  construct  algorithms  which  minimize  or  nearly  minimize  the  error. 

For  certain  classes  of  problems  it  shows  how  to  construct  algorithms  (linear 

I 

i optimal  error  algorithms)  which  enjoy  essentially  optimal  complexity  with 

l 

respect  to  all  possible  algorithms. 

The  existence  of  "strongly  non-computable"  problems  is  demonstrated. 

In  contrast  with  the  gap  theorem  of  recursively  computable  functions  we  show 
that  "every  monotonic"  real  function  is  the  complexity  of  some  problem. 
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CHAPTER  I 
BASIC  CONCEPTS 


Although  this  may  seem  a paradox,  all  exact  science 
is  dominated  by  the  idea  of  approximation. 

B.  Russell 


1.  INTRODUCTION 


This  is  the  first  of  a series  of  papers  constructing  an  information 
based  general  theory  of  optimal  error  algorithms  and  analytic  computational 
complexity.  Among  the  applications  are  such  traditionally  diverse  areas  as 
approximation,  boundary-value  problems,  quadrature,  and  nonlinear  equations 
in  a finite  or  infinite  dimensional  space. 

In  this  paper  we  deal  with  "general  information”.  Although  general 
information  can  be  used  to  solve  nonlinear  equations  (see  Section  15), 
"iterative  information"  is  typically  used  for  such  problems.  A theory  of 
optimal  error  algorithms  and  optimal  order  for  Iterative  information  is 
developed  in  Traub  and  Wozn iakowski  [77c]. 

The  general  theory  shows  us  how  to  construct  algorithms  which  minimize 
or  nearly  minimize  the  error.  For  certain  classes  of  problems  we  show  how 
to  construct  algorithms  (linear  optimal  error  algorithms)  which  enjoy  essen- 
tially optimal  complexity  with  respect  to  all  possible  algorithms.  The  fact 
that  optimal  algorithms  depend  only  on  the  "information"  used  permits  a 
great  simplification. 

Traditionally,  algorithms  are  derived  by  ad  hoc  criteria.  The  informa- 
tion based  theory  rationalizes  the  synthesis  of  algorithms.  For  illustra- 
tion we  consider  a relatively  simple  problem-univariate  numerical  integration. 

Gauss  quadrature  formulas  are  widely  used.  The  formula  is  derived  by 


t.it  rather  arbitrary  criterion  that  it  be  exact  for  all  polynomials  of  as 
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high  a degree  as  possible.  There  is  no  reason  to  think  this  optimizes  either 
error  or  complexity.  The  information  based  theory  yields  both  optimal  error 
algorithms  and  the  fact  that  these  algorithms  are  essentially  optimal  with 
respect  to  complexity  among  all  possible  algorithms.  It  turns  out  that 
the  optimal  error  algorithms  use  equi-spaced  abscissas  and  are  therefore 
not  Gaussian  formulas.  Furthermore  their  complexity  varies  inversely  as 
the  "regularity"  of  the  class  of  integrands.  It  has  been  observed  that 
the  numerical  integration  problem  cannot  be  solved  to  within  pre-ass igned 
error.  We  show  (Section  12)  that  if  we  mildly  restrict  the  class  0f 
integrands  we  can  always  solve  the  integration  problem  to  within  pre- 
assigned error. 

The  analysis  needed  to  characterize  and  construct  optimal  error  algo- 
rithms for  a particular  problem  area  can  be  difficult.  Note,  however,  that 
this  may  be  viewed  as  pre-conditioning  since  it  need  be  done  only  once. 

Among  the  major  questions  we  pose  and  at  least  partially  answer  are: 

1.  What  is  a lower  bound  on  the  error  of  any  algorithm  for  solving 
a problem  using  given  information?  See  Section  2. 

2.  In  general  is  there  an  algorithm  which  gets  arbitrarily  close 
to  this  lower  bound?  See  Section  2. 

3.  When  is  the  information  strong  enough  to  solve  a problem  arbitrarily 
closely?  See  Section  3. 

4.  What  is  the  most  relevant  information  for  solving  a problem? 

See  Section  6. 

If  we  are  willing  to  pay  enough  can  we  always  solve  a problem 
approximately?  See  Sections  7 and  9. 


5. 
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b.  For  linear  problems  is  there  always  a linear  algorithm  whose 
error  is  within  a constant  factor  of  having  optimal  error? 

See  Section  7. 

7.  If  we  assume  a Hilbert  space  setting,  what  characterizes  those 
problems  which  can  be  solved  arbitrarily  closely?  See  Section  7. 

8.  For  certain  application  areas  how  does  optimal  error  and  com- 
plexity depend  on  the  "regularity"  of  the  class  of  "problem 
elements"?  See  Sections  11,  12  and  13. 

Analytic  complexity  is  similar  to  abstract  and  concrete  complexity  in 
focusing  on  lower  bounds  of  problem  complexity.  Analytic  complexity  obtains 
its  characteristic  flavor  because: 

1.  Problems  cannot  be  exactly  solved  with  finite  complexity. 

2.  In  concrete  complexity,  problem  elements  are  assumed  given  and 
information  operators  do  not  explicitly  occur  in  the  theory 
because  they  are  identity  operators.  See  Section  3 for  examples. 

In  analytic  complexity  only  certain  "information"  about  the 
problem  element  is  obtainable. 

We  summarize  some  of  our  complexity  results  below. 

1.  Consideration  of  the  pre- image  set  of  an  "information  operator" 
gives  us  a general  adversary  principal  for  deriving  lower  bounds. 

2.  We  demonstrate  the  existence  of  "strongly  non-computable"  problems. 


See  Section  6. 


3.  We  show  the  existence  of  arbitrarily  hard  problems.  Furthermore 
"every  monotonic"  real  function  is  the  complexity  of  some  problem. 
See  Section  9.  This  may  be  contrasted  with  the  gap  theorem  of 
recursively  computable  functions  (Borodin  [72]). 

U.  For  certain  classes  of  problems  we  can  obtain  lower  bounds  on  the 
"inherent  problem  complexity"  for  all  possible  algorithms.  See 

Sections  7 and  10. 

5.  For  "linear  optimal  error  algorithms"  the  difference  between 
upper  and  lower  bounds  is  very  small.  See,  for  example.  Sections 
9,  10,  11. 

6.  We  perform  worst  case  analysis  over  all  problem  elements  in  a 
class.  In  contrast  with  other  recent  complexity  results,  we 
believe  Chat  the  worst  case  results  hold  for  "almost  all"  prob- 
lem elements . 

Although  we  feel  that  progress  has  been  made  towards  a general  theory, 
much  remains  to  be  done.  See  Section  16  for  a partial  list  of  extensions. 
Application  of  the  general  theory  to  various  problems  of  interest  will 
require  substantial  work  for  each  problem. 

We  summarize  major  concepts  and  results  of  this  paper. 

Sec tion  2 ■ We  define  d(31,s),  the  diameter  of  information  31  for  the  problem 
S,  and  r(3l,S)  , the  radius  of  information  31  for  the  problem  S and  show 

(Theorem  2.1)  that  r(3l,S)  provides  a best  possible  lower  bound  on  the  error 
of  any  algorithm  using  the  information  31.  We  define  interpolatory  algorithm 
and  show  (Theorems  2.2,  2.3)  that  any  interpolatory  algorithm  has  an  error 
which  differs  by  at  most  a factor  of  two  from  a lower  bound  on  the  error. 
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Wo  also  show  that  there  is  always  an  algorithm  which  is  arbitrarily  ciose 
to  the  lower  bound.  We  observe  (Corollary  2.2)  that  a problem  S using 
information  31  can  be  solved  to  within  an  error  whose  norm  is  at  most  e 
iff  r (31,  S)  < e. 

Section  3.  We  introduce  our  model  of  computation  and  the  major  complexity 
concepts.  In  particular,  we  define  primitive  operations,  permissible 
information  operators  and  algorithms.  We  define  the  e-complexity  of  a 
problem  in  a class  of  permissible  information  operators  Y as  the  complexity 
of  solving  problem  S to  within  e if  the  best  algorithm  and  the  best  informa- 
tion from  Y are  used. 

Section  4 . In  Sections  4-9  we  consider  linear  problems  and  linear  informa- 
tion. The  card  Inal  itv  (card  (31))  of  a linear  information  operator  is  defined 
and  we  show  (Lemma  4.2)  that  information  operators  with  finite  cardinality 
equal  to  n can  be  represented  by  n linearly  independent  linear  functionals. 

Section  5.  We  consider  problems  specified  by  a linear  solution  operator  S 
and  a linear  restriction  operator  T.  We  show  that  the  dependence  of  d(31,S,T) 
on  21  is  only  through  the  kernel  of  31.  We  define  index (S,T)  and  show 
(Theorem  5-2)  that  if  card(2!)  < index(S,T)  the  solution  cannot  be  ap- 
proximated to  within  e even  for  arbitrarily  large  e.  In  particular,  if 
index  (S,T)  = °°,  the  problem  cannot  be  solved  by  any  information  operator 
with  finite  cardinality. 

Section  6.  For  fixed  cardinality  what  is  the  most  relevant  information  for 
solving  a problem?  Formally,  the  n- th  minimal  diameter  of  information, 
d(n,s,T),  is  the  diameter  if  the  best  information  of  cardinality  at  most  n 
is  used.  Theorem  6.1  shows  d(n,S,T)  is  completely  determined  by  the 
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operator  ST  . A problem  is  e-non-computable  if  d(S,T)  = 1 im  d(n,S,T)  2 2e 

n-*00 

and  is  convergent  if  d(S,T)  =»  0.  We  show  that  d(S,T)  can  be  any  number. 

Section  7.  If  the  image  of  the  restriction  operator  is  a Hilbert  space  then 
the  problem  (S,T)  is  convergent  iff  ST  * is  a compact  operator.  This  implies 
(Corollary  7.1)  the  existence  of  linear  problems  which  cannot  be  solved  to 
within  arbitrary  e with  any  finite  number  of  linear  functionals.  In  a 
Hilbert  space  the  problem  of  most  relevant  information  of  cardinality  n is 
completely  solved  (Theorem  7.2). 

Section  8.  To  minimize  combinatorial  complexity  it  is  desirable  to  use 
1 Inear  algorithms.  We  construct  (Theorem  8.1)  linear  interpolatory  algorithms 
whose  error  is  within  a numerical  factor  of  optimality.  If  in  particular 
the  image  of  the  restriction  operator  is  a Hilbert  space  then  we  construct 
(Corollary  8.1)  a linear  interpolatory  optimal  error  algorithm. 

Section  9 . We  specify  our  model  of  computation  for  \the  linear  case.  We 
show  (Theorem  9.2)  there  exist  linear  problems  with  essentially  arbitrary 
complexity.  This  implies  (Corollary  9.1)  there  exist  arbitrarily  hard 
linear  problems  and  that  there  are  no  "gaps"  in  the  complexity  function. 

Sections  10-15.  We  apply  the  general  theory  to  a variety  of  problems.  See 
the  Table  of  Contents  for  a list.  We  confine  ourselves  here  to  relatively 
simple  problems  in  order  to  concentrate  on  ideas  and  avoid  overwhelming  the 
reader  with  technical  details. 

Section  16.  We  list  some  extensions  to  the  theory  which  will  be  reported 

e laewhere . 

Glossary.  For  the  reader's  convenience  a glossary  of  important  definitions 


and  symbols  is  provided. 
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2.  DIAMETER  AND  RADIUS  OF  GENERAL  INFORMATION 

Let  30  be  a subset  of  a linear  space  over  the  real  or  complex  field. 
Consider  a linear  or  nonlinear  operator  S such  that 

(2.1)  S: 

where  3^  i-s  a linear  normed  space  over  the  real  or  complex  field.  Let  e > 0 
be  a given  number.  Our  problem  is  to  find  an  e- approximation  x = x(f) , 
x £ 3^  to  a = S(f)  , i.e. 

(2.2)  ||  x-of||  < s 

for  all  f £ 3 • We  shall  call  S the  solution  operator,  f a problem  element 
and  v a soluLion  element.  We  shall  often  refer  to  S and  its  domain  3q  as 
the  problem  S . 

To  find  an  a- approximation  we  must  know  something  about  the  operator  S. 

Let 

(2.3)  51:  D„j  “■  3} 

be  an  information  operator  (not  necessarily  linear)  where  3q  ~ Dc,  C 3j  and 
3-j  is  a given  space.  21(f)  is  called  the  information  of  f.  For  most  prob- 
lems the  information  operator  -t  is  not  one-to-one  and  21(f)  does  not  uniquely 
define  the  solution  element  r/  =•  S(£).  Thus  there  may  exist  many  different 
problem  elements  f (E  3q  with  the  same  information. 

Let  f €-3,}-  Let 


(2.4)  V(f)  = {f:  21(f)  = 21(f)  and  f £ \ } 
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be  the  pre- image  set  of  y 3 ‘JUf),  V(f)  = '21  *(y)  . Note  that  V(f)  is  not 
empty  since  f £ V(f)  for  every  f £ Jt . Furthermore  let 

(2.5)  U ( f = [S(f)  : f € V(f)  } 

be  the  set  of  all  solutions  S(f)of  problem  elements  f which  share  the  same 
information  as  f,  U(f)  = S(D1  *(y)).  Then  knowing  only  *Jl(f)  it  is  impossible  to 
recognize  which  solution  element  y = S(f)  or  5 = S(f)  is  being  actually 
approximated  for  all  f £ V(f).  This  adversary  principal  can  be  schematized  as 
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(2.7)  rad(A)  = inf  sun  ji  a-a  jj 

a&3,  A ' 

is  called  Che  radius  of  A.  Roughly  speaking,  rad(A)  is  the  minimal  radius 
of  a "ball"  which  contains  A.  If  there  exists  c,  c £ ^ , such  that 

(2.8)  sup  j|  c-a  !|  =*  rad(A) 
a A 

then  c is  a center  of  A.  Note  that  c can  be  an  element  outside  A and  need 
not  be  unique. 


Definition  2 . 1 

We  shall  say  d(2t,S)  is  the  diameter  of  information  ?1  for  the  problem  S 


iff 


(2.9)  d(21,S)  = sup  diam(U(f))  f=  sup  sup  ||  S (f ) - S (f ) | j\  . 

fe30  V fGv(f)  / 

We  shall  say  r(21,S)  is  the  radius  of  information  ?1  for  the  problem  S iff 


(2.10)  r(2t,S)  = sup  rad(U(f))  (=  sup  inf  sup  ||  a-S(f)|| 


££3o 


fG30  36  32  f6V(f) 
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It  is  obvious  that 

(2.11)  d (2l,S)  ^ 2r(2l,S)  . 

Furthermore  if  U(f)  has  a center  c(f)  for  every  f 6 3q  ancl  U(f)  is  symmetric 
with  respect  to  c(f),  i.e.  u + c(f)  £ U(f)  implies  -u  + c(f)  £ U(f) , then 

(2.12)  d(3l,S)  = 2r(5l,S). 

For  many  21  and  S it  is  much  easier  to  compute  the  diameter  d(21,S)  than  the 
radius  r (21, S)  . 


We  shall  show  that  the  radius  r(2l,S)  is  a lower  bound  on  the  error  of 
any  algorithm  for  solving  y = S(f)  . By  an  a Igor ithm  we  mean  an  operator 
'{>:  -1(3q)  (See  also  the  definition  of  "nermissible  algorithm"  in  Sec- 

tion 3.)  We  are  interested  in  algorithms  which  approximate  rt  = S(f).  Let 
5(»,S)  be  the  class  of  ai  i such  algorithms.  Since  -p("l(f))  *»  rpC^Cf))  for 
all  f V(f> , s has  to  approximate  any  element  of  the  set  U(f)  =*  S(^"^(y)). 


Figure  2 


Definition  2.2 

We  shall  say  e(;p)  is  the  error  of  algorithm  y iff 

(2.13)  e(cp)  ■ sup  ||  cp('jj(f))-S(f)  ||  . ■ 

Note  that  (2.13)  can  be  rewritten  as 

(2.14)  e (cp)  = sup  sup  ||  cp(^(f))-S(f)  ||  = sup  sup  ||  9(2l(f  1) -a||  . 

f£30  fi£V(f)  fe^  SeU(f) 


2.5 


It  is  intuitively  obvious  that  the  radius  r(’„2,S)  is  a lower  bound  on  the 
error  of  any  algorithm.  A formal  proof  is  provided  by 

Theorem  2 . 1 

For  any  algorithm  -p,  9 € 5(31, S)  , 

(2.15)  e (cp)  ^ r (21, S)  . ■ 

Proof 

Let  f £ Then  due  to  (2.7)  and  (2.14)  we  get 

rad(U(f))  ^ sup  [j  cp(Tl(f)  )-5;|  £ e (*p)  . 

3eT(f) 

Thus  r(n,S)  = sup  rad(U(f))  ^ e(f p)  which  proves  (2.15).  H 


This  generalizes  Theorem  4 in  Micchelli  and  Rivlin  f77]  where  S and  71 
are  assumed  to  be  linear. 

We  define  "interpolatorv  algorithms'1  and  show  they  are  within  a factor 
of  2 of  the  radius  r(3l,S). 

Def init ion  2 . 3 

An  algorithm  9^,  9^  € 5(12,5),  is  an  interpolatorv  algorithm  iff 
(2.16)  9I(,-'Kf))  = S(f) 

for  some  f £ V(f)  . ® 

This  means  that  knowing  the  information  31(f)  one  finds  a problem  element 
f (which  always  exists)  which  has  the  same  information  as  f , f f V(f) , and 
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3 = S(f)  is  proposed  as  an  approximation  to  'v  = S(f) . In  practice  f is  chosen 
to  be  "simpLer"  than  f.  Note  that  £ U(f).  In  some  cases,  an  assumntion 

•m 

how  to  choose  a unique  f is  added.  Examples  of  interpolatory  algorithms 
are  known  for  such  problems  as  nonlinear  equations,  approximation  and 
quadrature . 

Theorem  2 . 2 

For  any  interpolatory  algorithm  r3^ , £ 5('Jl,S), 

(2.17)  eOa1)  * d(Sl,S)  *2r(2l,S).  ■ 

Proof 

Take  any  f £ 3q*  Then 

II  9X(!n(f))-S(f)  ||  * ||  S(f)-S(f)  ||  S d (31, S) 

. ince  f £ V(f) . Taking  the  supremum  with  respect  to  f we  get  (2.17).  ■ 

We  seek  "optimal  error  algorithms"  which  are  defined  as  follows. 

Definition  2.4 

We  shall  say  e(7t,S)  is  the  optimal  error  iff 

(2.18)  e (01, S)  ■ inf  e(®)  . 

<pet(9t.s) 

oe  oe  _ 

We  shall  say  rc  , cp  fc  , is  an  optimal  error  algorithm  iff 

(2.19)  e('5°e)  = e(3l,S)  . ■ 

Combining  Theorems  2.1  and  2.2  we  see  that  any  interpolatory  algor ithm  is 
nearly  an  optimal  error  algorithm. 


A 
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► 


Corol larv  2 . 1 


For  any  interpolatory  algorithm  'o' , <3  ?(?!,S),  with  the  convention  -pj  = 1 , 


<2'20’  1 s2- 


We  now  prove  that  the  optimal  error  e(T!,S)  is  equal  to  the  radius  r(2l,S). 

Theorem  2.3 

(2.21)  e (31, S)  = r(«t,S) . ■ 

Proof 

Let  5 > 0 be  an  arbitrary  number.  Define  an  algorithm  ©.  as  follows. 

o 

Let 


(2.22)  -o.  («l(f  > ) = c.(f) 

* c 

where  ||c,(f)-ci||  £rad(U(f))  + 5 for  all  5 <3  U(f).  Thus  c.(f)  is  almost  a 
center  of  U(f).  Then 

e (21, S)  * e('o J = su?  J|  S (f> (21(f) > ||  - 

= sup  sup  Jj  |j  z sup  rad (l’ (f ) ) + 5 = 

5tu(f)  fe^0 

= r (?l,S)  + 5. 

Since  6 is  arbitrary,  e(2l,S)  ■£  r(2t,S).  ftie  to  Theorem  2 . 1 we  know  that 
e(21,S)  ^ r (2l,S)  which  proves  (2.22).  ■ 

See  Micchelli  and  Rivlin  [77]  where  a similar  result  is  established  for 
21  and  S linear.  Theorem  2.3  motivates  using  a center  c(f),  if  it  exists, 


2.8 


as  an  approximation  to  ct  ~ S(f).  Suppose  that  U(f)  has  a center  for  any 

f ^ 30- 


Definition  2.5 

c c 

An  algorithm  ® , ts  6 ?(71,S)  is  a central  algorithm  iff 
(2.2  3)  -pC(^(f>)  = c(f) 

where  c(f)  is  a center  of  U(f).  ■ 

Theorem  2.4 

Any  central  algorithm  is  an  optimal  error  algorithm,  i.e., 


(2.24)  e ("3  ) = r(?t,S) 


Proof 


Note  that 


e(-jC)  = sup  ||  S(f)-'DC  (21(f))  ||  = sup  sup  ||  5-c(f)|| 

3eu(f) 

= sup  rad(U(f))  = r(2l,S).  ■ 

ffc3o 

As  we  shall  see  in  Section  8 an  interpolatory  algorithm  may  turn  out 
to  be  an  optimal  error  algorithm. 

Recall  we  wish  to  find  an  e-approximation  to  j = S(f)  for  all  f6  , 
i.e.,  to  find  x(f)  such  that  ||x(f)-a||<  e.  Due  to  Theorem  2.3  we  get 

Corollary  2.2 

It  is  possible  to  find  an  e- approximation  to  ty  m S(f)  for  all  f 6 iff 


a. 
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(2.25)  r(5t,S)  < e.  ■ 

We  wish  to  stress  that  an  information  operator  52  has  to  be  defined  in 
such  a way  that  52(f)  is  "computable"  for  every  f € 3q*  This  rules  out  many 
operators  as  "permissible"  information  operators.  For  instance,  let  21(f)  = f 
be  the  identity  operator  I.  Then,  since  I is  one  to  one,  r(I,S)  = 0 for  any 
solution  operator  S.  However  51(f)  5 f is  "computable"  iff  f can  be  repre- 
sented by  a finite  dimensional  vector,  i.e.,  [2^  is  a finite  dimensional 
space.  As  a second  example,  consider  51(f)  = S(f) . Then  r(S,S)  = 0 but  for 
most  problems  S(f)  is  not  "computable"  and  51  = S is  not  a "permissible" 
information  operator.  See  Sections  3 and  9 for  a precise  definition  of  our 
model  of  computation.  Examples  of  computable  operators  will  be  found  in 
Sections  10-25. 

Examp  1 e 2.1 

To  illustrate  the  above  concepts  we  consider  the  following  problem. 

Tot  3^  — C [0,1]  be  the  class  of  n times  differentiable  functions  of  one 
variable,  n ^ 1.  Define 

(2.25)  S(f)  = f 

that  is,  S = I.  Mote  that  (2.25)  is  a formulation  of  the  approximation  problem. 
Let 

f £ " If:  f £ 3,  and  max  |~f(n)(x)|  S l}. 

0«tsi 

Consider  the  information  operator  5t  given  by 


(2.26)  51(f)  = [ f (tj)  ,f  (t2) f (tn)  ] 


2 . 10 


for  some  distinct  points  t ^ £ [0,1],  This  means  that  we  want  to  approximate 

n 

f from  3q  knowing  only  the  values  of  f at  n points.  Let  'jj(t)  = ( ) ( t- 1 . ) . 

1=“  1 1 

Then  f 6.  V(f)  implies 

f (t)  - f (t)  + g(t)ui(t) 


where  g is  the  nth  divided  difference  of  f-f  and 
is  easy  to  snow  that  d(2l,S)  = 2r(-l,S)  and 

(2  .27)  r(3t,S)  =■  j|  tu||  2 2/4°. 


max  |g(t) | £ 2.  It 
0£t£l 


Furthermore  (2.27)  holds  with  equality  for  information  21  of  the  form  (2.26) 

* tt  i- 1 

with  t^  * t.  3 2 cos(—  + — tt)  which  are  the  zeros  of  the  Chebyshev  poly- 
nomial T (■qT  t+ 1)  ) . See  Section  11. 
n 2 

If  e 2 24°  we  can  find  an  e- approximation  to  a 3 S(f)  = f for  all 

f G using  the  information  operator  21  w’ th  t.  = t,.  For  e < 2/4°,  the 
l)  ii  ’ 

information  operator  21  of  the  form  (2.26)  does  not  supply  enough  information 
to  find  c-approximations  for  any  t^.  ® 

We  conclude  this  section  with  a historical  note.  The  ideas  presented 
here  have  been  implicitly  used  by  a number  of  people  for  a particular  prob- 
lem or  a class  of  problems.  The  key  point  was  always  to  find  a problem 
element  f which  shares  the  same  information  as  f and  the  distance  between  S(f) 
and  S ( f ) was  an  inherent  error  of  any  algorithm.  See  among  others  Winograd 
[76]  who  introduced  a very  general  "fooling"  technique  and  showed  its  impor- 
tance for  a number  of  problems,  Micchelli  and  Rivlin  [77]  who  considered 
linear  operators  S and  <J1,  and  Wozniakowski  [75]  who  introduced  the  concept 
of  order  of  information  for  the  solution  of  nonlinear  operator  equations. 
Brent,  Winograd  and  Wolfe  [74],  Kung  [75],  Kacewicz  [75,  76a,  76b ],  Meersman 
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[76a, 76b],  Traub  and  Wozn iakowski  [76a],  Wozniakowski  [76] 
the  solution  of  nonlinear  equations  and  WerschuLz  [77a,  77 
maximal  order  of  numerical  integration  and  differentiation 


all  considered 
i]  dealt  with  the 


r 


3.1 


3.  COMPLEXITY  OF  GENERAL  INFORMATION 


We  present  our  model  of  computation  which  consists  of  a set  of  primi- 
tive operations,  permissible  information  operators  and  permissible  algo- 
rithms. In  what  follows  we  shall  use  the  words  cost  and  complexity  inter- 
changeably. Context  will  make  it  clear  whether  we  mean  algorithm  complexity 
or  problem  complexity. 

Model  of  Computation 

(i)  We  assume  that  the  computations  are  performed  on  a random  access 
machine.  Let  p be  a primitive  operation.  Examples  of  primitive 
operations  include  arithmetic  operations,  the  evaluation  of  a 
square  root  or  of  an  integral.  Let  comp(p)  be  the  complexity  of 
p;  comp(p)  must  be  finite.  Suppose  that  P is  a given  collection 
of  primitives.  The  choice  of  P and  comp(p) , p 6 P,  are  arbitrary 
and  can  depend  on  the  particular  problem  being  solved. 


(ii)  Let  21  be  an  information  operator.  We  say  that  21  is  a permiss ible 
information  operator  with  respect  to  P if  21(f)  can  be  computed 


by  a finite  number  of  primitive  operations  from  P for  all  f £ 3q • 
Let  comp(21(f))  denote  the  information  complexity  of  computing 
21(f)  . We  assume  that  if  21(f)  requires  the  evaluation  of  prirai- 

k 


tives  p1,p2>...,pk  then  comp(2l(f)) 


comp(pt) . 


(iii)  Let  be  an  algorithm  which  uses  the  permissible  information  21. 
To  evaluate  ,p(2l(f))  we: 


3.2 


(a)  compute  y = 31(f), 

(b)  compute  cp(y)  . 


The  complexity  of  computing  y is  given  by  (ii).  We  say  that  'o 
is  a permissible  algorithm  with  respect  to  P if  ®(y)  can  be 
computed  by  a finite  number  of  primitive  operations  from  P for 
all  y = 21(f),  fO^g-  Let  comp(rp(y))  be  the  combinatory 
complexity  of  computing  y.  We  assume  that  if  ®(y)  requires  the 

i 

evaluation  of  primitives  q ^ ,q9 , . . . ,q  . then  comp(®(y))  = ^ comp(qJ. 

i=l 


Remark  3 . 1 

Let  31  be  a permissible  information  operator.  This  means  that  31(f)  can 
be  computed  from  the  set  of  primitives  P.  Often  there  exist  many  different 
algorithms  for  computing  31(f)  and  the  optimal  computation  of  31(f)  can  be 
treated  as  a subproblem.  However,  we  assume  that  an  algorithm  (possibly  not 
optimal)  for  the  computation  of  31(f)  is  defined  by  a "user". 


Example  3 . 1 


1 

Suppose  we  wish  to  approximate  S(f)  = f(t)dt  where  f’  is  a piecewise 

\ 2 5 

continuous  scalar  function  and  (f’(t))  dt  £ 1.  Define  two  sets  of  primi- 

0 

tives,  = ;.the  evaluation  of  an  integral]  and  P^  = {arithmetic  operations, 
the  evaluation  of  a function].  Note  that  31(f)  = S(f)  is  permissible  with 
respect  to  P^  and  not  permissible  with  respect  to  P^ . Of  course  r(S,S)  * 0. 

However,  S is  a primitive  only  in  P^ . An  example  of  a permissible  information 
operator  for  ?2  is  31(f)  = [ f ( 1 1 ),  f (t? ),...,  f ( tn>  ] for  equally  spaced  t.6  [0,1]. 
It  is  shown  in  Section  12  that  r(31,S)  =>  0(l/n).  ■ 

We  showed  in  Section  2 that  a necessary  and  sufficient  condition  for 
finding  an  e-approximation  to  a = S(f)  is  r(3l,S)  < e.  If  r(31,S)  2 e then  the 


J 


3.3 


information  operator  21  does  not  supply  sufficient  information  to  solve  the 
problem.  We  say  that  the  problem  S with  an  information  operator  21  is 
c- non- computable  if  r(21,S)  ^ e.  If  21  is  permissible,  r(21,S)  < e,  and  there 
exists  a permissible  algorithm  cp  such  that  e(cp)  < e,  then  the  problem  S with 
21  is  called  e-ccnnputable  with  respect  to  P. 

Suppose  then  that  r(2l,S)  < e for  a permissible  21  and  assume  that  $(e)  , 
the  class  of  all  permissible  algorithms  for  which  e (cp)  < e,  is  non-empty. 

We  want  to  derive  lower  and  upper  bounds  on  the  complexity  of  finding  an 
e-approximation  using  any  r c6  $(e). 

Since  the  set  of  primitives  P is  fixed,  we  do  not  mention  the  dependence 
of  complexity  on  P.  Let  cp  6.  $(e)  . Then  the  complexity  of  an  algorithm  cp  is 
defined  by 

(3.1)  comp  (cp)  = sup  (comp  (21(f)  ) + comp(ro(^(f ) ) ) ) . 

We  define  optimal  complexity  algorithm  (Definition  3.1)  and  inherent  com- 
plexity of  a problem  (see  Definition  3.2). 

Definition  3 . 1 

We  say  comp  (21, S , e)  is  the  e-complexity  of  the  information  21  for  the 
problem  S iff 


inf  comp(cc)  , if  r(9l,S)  < e 
ci)t*(e) 


(3.2)  comp(ni,S,e)  = "N 


, otherwise 


We  say  cc°C , rp°C  € *(e)  is  an  optimal  complexity  algorithm  iff 


oc 


(3.3)  comp(<®  ) “ comp(!Jl,S , e) 


3.4 


Let 


(3.4)  comp  (71)  = sup  comp  (71(f)) 

be  the  information  complexity  of  71.  Suppose  that  the  combinatory  complexity 
of  every  algorithm  ©,  cp  €.  5(e),  for  71(f)  such  that  comp(71(f))  = comp(71)  is 
bounded  below  by  m(71)  . More  precisely,  let 

(3.5)  m(71)  = inf  sup  comp  (cp(7l(f ) ) ) . 

©£*(£)  f:  comp  (71(f)  ) =comp (71) 

In  general,  m(7l)  depends  on  the  total  number  of  "independent  pieces"  of 
information  71.  See  Section  4 and  9 where  the  "cardinality"  of  information  71 
is  introduced  and  its  influence  on  the  combinatorial  complexity  of  © is 
shown.  For  linear  problems,  as  we  shall  see  in  Sections  8 and  9,  it  is  pos- 
sible to  find  optimal  algorithms  whose  combinatory  complexity  is  proportional 
to  the  "cardinality"  of  71. 

From  (3.4)  and  (3.5)  we  get 

(3.6)  comp(7l,S,e)  2 comp(71)  + m(7l)  . 

Furthermore  if  there  exists  © £ 5(e)  such  that  comp  (©(7l( f) ))  « comp(71)  for 
all  f € 3q  then 

(3.7)  comp(7l,S,e)  - comp (71) 

Equations  (3.6)  and  (3.7)  motivate  our  interest  in  comp(71). 

Suppose  that  e tends  to  zero.  Then  if  r(71,S)  > 0,  the  fixed  information 
71  is  weak  for  the  problem  S for  sufficiently  small  e.  However  in  many  cases 
we  can  choose  a permissible  information  operator  71  = 71(e)  such  that 
r (51(e), S)  < e and  solve  our  problem  using  the  information  71(c). 


3.5 


Let  Y be  a class  of  permissible  information  operators  such  that 

(3.8)  inf  r (71, S)  = 0. 

UtfcY 

Note  that  (3.8)  means  that  we  can  solve  the  problem  S for  any  e using  a 
suitably  chosen  information  operator  from  Y. 

Definition  3.2 

We  shall  say  comp (Y, S , e)  is  the  c-complexity  of  the  problem  S in  the 
class  Y iff 

(3.9)  comn ( Y , S , e)  =*  inf  comp ("l, S , e)  . ■ 

2t€Y 

Note  that  comp(Y,S,e)  is  a nonincreasing  function  of  e.  We  shall  prove 
ia  Section  9 that  comp(Y,S,«)  can  be  an  essentially  arbitrary  nonincreasing 
function  of  c. 

Our  setting  is  sufficiently  general  that  it  includes  problems  for 
which  information  operators  do  not  play  a role.  Examples  are  combinatorial 
problems  and  such  problems  of  linear  algebra  as  matrix  multiplication  and 
the  direct  solution  of  linear  systems.  For  such  problems  the  information 
operator  is  the  identity  operator  21(f)  = f where  f belongs  to  a finite 
dimensional  space.  Furthermore  the  information  complexity  comp(I)  = 0 since 
there  is  no  cost  in  computing  21(f)  * f.  Note  that  r(I,S)  “ 0 for  any  prob- 
lem S because  71  » I is  one  to  one.  Thus  we  can  define  Y = {i}.  Typically 
we  seek  the  exact  solution  a * S(f)  ; thus  e =*  0 . The  complexity  is  given  by 

comp ( I , S , 0)  = inf  comp(m(f)). 
cp:  e(cp)“0 

Therefore  in  algebraic  and  combinatorial  complexit-y  we  seek  an  algorithm 
which  finds  a m S(f)  and  has  minimal  combinatory  complexity. 
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Example  3.1.  Matrix  Multiplication 

Let  f = [A,B]  where  A and  B are  nxn  matrices.  Then  the  matrix  multi- 
plication problem  may  be  formulated  as  S(f)  = A*B.  Let  Ul( f ) = f = [A,B]. 

This  means  that  all  coefficients  of  A and  B are  known  and  we  seek  an  algo- 
rithm with  minimal  combinatory  complexity  which  yields  the  matrix  x(f)  * A*B. 

If  the  cost  of  each  arithmetic  operation  is  taken  as  unity,  then 

2 8 

c^  £ comp  ( I , S , 0)  * c7np,  « = log7  7 

for  some  positive  constant  c^  and  c0.  The  actual  value  of  comp(I,S,0)  is 
unknown . ■ 

Example  3.2.  Sorting 

Let  f = ff,,f_ f ] where  f . €.  D and  D is  an  ordered  set.  Define 

L 1 2 nJ  l 

S(f)  = [ft  ,f,  ] 

1 n 

where  f sf.  ^ ...  if.  and  i ,...,i  is  a permutation  of  l,...,n.  Then 

1 4 i-O  ^ ^ 

12  n 

the  sorting  problem  may  be  formulated  as  or  = S(f).  Let  21(f)  = f and  e = 0. 

We  seek  an  algorithm  which  finds  S(f)  with  minimal  cost  where  the  cost  is 
taken  as  the  number  of  comparisons.  The  complexity  of  this  problem  satisfies 

comp(I,S  ,0)  = (n  log2  n)(l+o(l))  ■ 

Recently  there  has  been  an  interest  in  finding  e-approximations  to  the 
solutions  of  algebraic  and  combinatory  problems.  For  some  problems  the 
complexity  comp(I,S,e)  is  significantly  smaller  for  positive  e than  comp(I,S,0). 
Examples  may  be  found  in  Garey  and  Johnson  [76]. 


3.7 


Exanple  3.3,  Polynomial  Zero 

Let  f - (ag*3! an  ] where  the  a.  are  the  coefficients  of  an  nth 

degree  polynomial,  P(x)  . Let  P(a)  = 0,  Cr  = S(f)  . For  algorithms  which 
require  knowledge  of  the  a.,  3^  (f)  - f.  On  the  other  hand,  there  exist 
iterative  algorithms  requiring  only  that  P and  P'  can  be  evaluated  at  any 
point  and  3L(f)  = [x,P(x)  ,p'  (x)  ]. 
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CHAPTER  II 
LINEAR  THEORY 


We  assume  that  the  soLution  operator  Sj  the  "restriction  operator  T 
and  the  information  operator  21  are  linear. 

4.  CARDINALITY  OF  LINEAR  INFORMATION 


Let  21;  3^  03  be  a linear  information  operator  where  i-s  a linear  space, 

ker  21  = {f : 2Uf)  = 0]  be  the  kernel  of  31.  We  shall  prove  in  Section  5 that 
the  dependence  of  the  diameter  of  information  on  31  is  only  through  the 


» 1 r Cf> 

x.ernei.  01  .1. 
operators  wi 
Let  2^: 
is  a linear 


rr-n  • . . a.  — ....  -1 1 J *-  nV>  UnKtnnn  hm  i n f nrmfl  f*  1 nn 
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th  the  same  kernel. 

3 3^  and  2^:  3^  - 3^  be  two  information  operators  where  3^ 

space  not  necessarily  equal  to  ^3 • 


Let 


Definition  4 . 1 

We  shall  say  21^  is  contained  in  21^  (briefly  21^  C 21?)  iff  ker  21^  c ker  21^. 

We  shall  say  21^  is  equivalent  to  21^  (briefly  31^  h:  310)  iff  ker  21^  = ker  21^.  B 
Note  that  "ar"  is  an  equivalence  relation. 

We  want  to  show  that  31^  C 21  can  be  characterized  by  the  rank  of  a 
certain  matrix.  We  first  briefly  recall  some  facts  on  linear  spaces.  Let  A 
be  a linear  subspace  of  3^  Then  there  exists  a linear  subspace  A-1  of  31 
such  that 

(4.D  31  = a e a-1-. 


4.2 


In  general,  is  not  uniquely  defined.  However,  if  "T  is  a Hilbert 
space  and  A is  closed,  then  there  exists  a unique  orthogonal  A1  to  A such 
that  (4.1'  holds.  In  either  case  is  isomorphic  to  the  quotient  space  \ A 

(4.21  codia  A ==  dim  A^"  = dim  3j  A. 

The  space  A^"  is  called  an  algebraic  complement  of  A in  the  space  • 

Let  L,,L~,...,L  be  linearly  independent  functionals.  By 
12m 

(4.3)  - [L1,L2,...,Lra]t 

we  mean  2l^(f)  = [L^(f) ,L0 (f) , . . . ,Lm(f) ] where  "t"  denotes  the  transpose 

of  a vector  and  f £ j\^. 

Lemma  4 . 1 

Let  2^  = [L1,L2  , . . . ,Ln]t  and  2t0  = [ Ln+ x *Ln+'<  > • • • »Ln+k  ] C be  information 
operators . 

(i)  21^  c 2^  iff  k a n and  there  exists  a nxk  matrix  M such  that 

21  ^ = M-V,  and  rank  M = n. 

(ii)  Let  k = n.  Then  2^  C 21  iff  21  X 21  . ■ 

Proof 

(i)  Suppose  that  ker  21  — ker  21  . Let  3,  = ker  21  6 (ker  2l0)"^  and 

(ker  212)^  = 1 in  (5^  , ?2 whejre  Ln+j(?^)  = and  5 ^ j denotes  the 

Kronecker  delta.  Then  f = f„  + L , .(f)?.  where  f„  £ ker  21„  and  f £ \ . 

0 _ n+ 1 i 0 2 1 

i“l 

Since  fg  ker  21  we  get 


and 


A. 3 


(4  .A)  1,(0 


L .(f)L.(-:.)  for  j - 1,2, 
n+x  J r 


.n. 


x5*  1 


This  yields  S,  - >«,  -ith  M - >•«  <*«  */  ' “»  V 

where  t.(V  ’ V S-  ' * \ ^ •*»  £°r  1 ’ U2 “• 

, . 1 “ the  "xn  idc"tl,:y  ”erl,t'  Thls 

imp  lies'  the  t rink  M - a end  completes  this  pert  of  the  proof.  Soppose  sow 

that  ft.  - Then  h 6 her  t,  implies  5,0.)  - M^OO  ■ 0 which  yields 


* 1 


*’  1 ' 


(ii)  Suppose  that  21  ^ 


Due  to  the  first  part  of  Lemma  A.l  we 


t m 3 ^ where  the  n/n  matrix  M is  nonsingular.  Then  % = M \ whxch 

get 

• ..Mlr,ind5S  «L.  The  second  part  of  (ii)  is  trivial.  Hence 
implies  -t0  - anu  2 

1 

Lemma  4.2  is  proven. 

W.  now  she  that  any  information  operator  5 where  „ - codim  her  ft  is 
finite  „y  »e  ..presented  by  n linearly  independent  linear  fenctionals. 


Lemma  A . 2 

Lee  «l  be  an  information  operator  and  n - codim  ker 
exist  Linearly  independent  linear  functionals  L^L,,....^  such  that 

,t 


cc.  Then  there 


”1  ^ where  21  ^ = [1^  ’^2  ’ ’ ’ * ,En 


roof 

Let  (ker  %L  = lin(§1,?2. . • • . V Every  ^ 

station  f - fQ  + ^ Vf)5i  where  f Q 6 ker  21  and  L, ,1^ , . . . , Ln  are 

linearly  independent*! inear  functionals.  Since  ker  "l  - (f:  Lt(f)  °* 
x - 1,2, . . • ,n]  ” ker  we  get  21  - ^ 


4.4 


Lemma  4.2  states  that  an  information  operator  where  n = codim  ker  21  is 
finite  is  equivalent  to  an  information  operator  generated  by  n linearly 
independent  linear  functionals.  Observe  that  to  know  21^(f;  one  has  to 
evaluate  n linear  functionals.  This  suggests  the  following  definition  of 
the  cardinality  of  21. 

Definition  4.2 

We  shall  say  that  card(2l)  is  the  cardinality  of  the  information  21  iff 

(4.5)  card(21)  = codim  ker  21.  ® 

We  shall  prove  in  Section  5 that  unless  the  cardinality  of  the  informa- 
tion 21  is  sufficiently  large,  the  diameter  of  information  is  infinity  and 
the  problem  cannot  be  solved  with  this  information. 

To  illustrate  the  concept  of  cardinality  we  consider  two  examples. 


Example  4 . 1 


Let  21  = [L^ .Lj » • • • »L  ]t-  From  the  above  considerations  easily  follows 
that  card (71)  £ n and  card(21)  - n iff  L ,L2  , . . . ,Ln  are  linearly  independent. 


Example  4 . 2 


Let  f:  D clm  -» (£a  be  a k times  differentiable  function.  Let 


21(f)  = f'  ' (x)  for  x €.  D. 


Note  that  f(x)  =■  [ f . (x)  , f (x)  , . . . , f (x)  ]C  where  f • D - (T-1  is  a scalar 

i Z m j 


(nH-k-  l\ 

k J linearly  independent  func- 

tionals  of  the  form  L(f)  - where  x = [x,  ,x  , ...,x  l*"  and  p a 0, 

p . p 12m  i 

Ox/.-.dx  m 
1 m 


inearly  independent  func- 


).  + P-  + ...  + p “ k.  This  yields 
12  m 


4.5 


card  (71)  = ^ ' 

If  71(f)  = [f  (x)  ,f  ’ (x) f*^n  ^ (x)  ] is  the  so-called  standard  information, 

then 

“rt<*  * I''  <\k")  ■ -te1)  • 

k“0 

This  shows  the  dependence  of  cardinality  on  the  dimension  of  the  spaced,™.  ■ 

We  end  this  section  by  showing  that  there  is  a one-to-one  correspon- 
dence between  information  operators  and  subspaces  of  3^* 

Let  be  an  information  operator  with  card(7l)  ■ n.  Then  AC??)  ==  ker  71 
has  codimensionality  equal  to  n.  Furthermore  71^  — 71^  implies  A(Tl^)  = AC^^)  • 
We  now  show  that  the  converse  statement  is  also  true. 


Lemma  4 . 3 

Let  A be  an  arbitrary  linear  subspace  of  3^ 
Then  there  exists  a unique  (up  to  the  equivalence 
operator  71  with  card (71)  * n such  that  A = ker  71. 


such  that  codim  A =■  n. 


Proof 

Let  3j  = A €•  A*^  where  A-*-  = lin(|^,^2 ? ).  Then  f 

where  f £ A and  L.(^.)  * 6...  Define 
0 i'Jj  lj 


f0  + 


^(f)^ 


i=l 


71 


[L1,L2 


Since  L^, L^ , . . . .L^  are  linearly  independent,  card(7l)  = n and  ker  71  = A . To 
prove  the  uniqueness,  observe  that  if  A = ker  71^  = ker  71  o then  71^^  71^. 


This  completes  the  proof. 


5.1 


5.  INDEX  OF  A LINEAR  PROBLEM 

We  consider  in  this  section  linear  information  operators  for  the  solu- 
tion element  ~n  = S(f)  where  S:  % is  a linear  operator  and  3q  is  de- 

fined as 

(5.1)  ^ = If  £ : II  Tf  ||  * 1} 

where  T : A "*  ^ ~ T ( 3^ ) Is  a linear  operator  and  % is  a linear  normed 

1 *4  1 <4 

space  over  the  real  or  complex  field.  We  shall  call  T the  restriction  operator. 
This  means  we  want  to  find  an  e-approximation  to  the  solution  n = S(f)  for 
all  f such  that  J|  Tf  ||  £ 1. 

To  stress  the  dependence  on  T we  shall  replace  S by  (S,T)  in  all  basic 
definitions.  For  instance  we  shall  refer  to  the  problem  fS,T),  the  diameter 
d('-'t,S  ,T)  , etc.,  where  'It  is  a linear  information  operator. 

Without  loss  of  generality  we  choose  a bound  jj  Tf  ||  £ 1 instead  of  j|  Tf||  s c 
for  a positive  constant  c.  Indeed,  let  T^  = ^T.  Then  |j  T^f|j  S 1 is  equiva- 
lent to  , j Tf  | J £ c.  It  is  easy  to  observe  that  d(71,S,T)  = cdCTl.S.T^)  and 
all  estimates  on  complexity  are  linear  in  c. 

We  now  show  that  the  dependence  of  d(71,S,T)  on  71  is  only  through  the 
kernel  of  71. 

Lemma  5 . 1 

(5.2)  d(71,S,T)  - 2 sup  ||  Sh|| 

htV(O) 

where  V(0)  = ker  71  fl  3g.  (See 


(2.4)  .) 


5.2 


Proof 

Set  c = 2 sup  ||  Sh  ||  . Let  f £ 3n  and  f £.  Vff).  Then  h = -^ff-f)  €.  ker  21 
h£V(0)  2 

and  ||  Th|j  s 1.  This  yields 

||  S f - S f 1 1 = 2 1 1 Sh  ||  Sc. 

Taking  the  supremum  with  respect  to  f and  f we  get  d(2l,S,T)  Sc.  To  prove 

the  reverse  inequality,  let  h € V(0) . Set  f = h and  f = -h.  Then  f £.  V(f)  = V(0) 

and 

2 ||  Sh  ||  = ||  Sf-Sf  ||  S d (31,  S , T)  . 

Thus  c S d(2t,S,T)  which  completes  the  proof.  ■ 

From  Lemma  5 . 1 we  immediately  get  the  following  Corollary. 

Corollary  5 . 1 

If  21  c3l2  then  d(212,S,T)  s d^.S.T), 

If  31  X 31  then  d(3l2,S,T)  = d^.S.T).  ■ 

In  Section  2 we  showed  that  the  radius  r(3l,S,T)  is  the  intrinsic  error 
of  the  information  21  and  the  problem  (S,T).  Due  to  Lemma  5.1  we  get 

c s r(2l,S,T)  s 2c 

where  c * sup  ||  Sh  j|  . We  now  show  when  r(21,S,T)  = c. 
h€V(0) 

Lemma  5 . 2 

If  for  any  f £ 3q  there  exists  h^  £ ker  21  such  that  Th^  = Tf  then 


r (21,  S ,T) 


sup  ||  Sh 
h£V(0) 


5.3 


Proof 

Let  a = S ( f-hg)  . Then  for  any  f = f + h£  V(f),  h€  ker  71,  we  get 

j|  a—  Sf  1 1 = ||  Sz  j where  z 3 + h £ ker  71.  Since  Th^  = Tf,  ||  Tz||  = 

ij  T(z-h^)+Tf|j  =■  ; Tf|j  1.  From  (2.10)  and  Lemma  5.1  we  get 

r (71,S  ,T)  s sup  sup  j|  a-sT  j|  = sup  ||  Sz  ||  = ^d (71, S ,T)  . 
f€V(f)  z£V(0) 

Since  r(71,S,T)  s^d(71,S,T)  for  any  71,  s and  T,  Lemma  5.2  is  proven.  fl 

We  want  to  examine  when  the  diameter  d(71,S,T)  is  equal  to  infinity. 

(Of  course,  d(7l,S,T)  =*  + 03  implies  r(71,s,T)  = + °°.)  We  begin  with 

Theorem  5 . 1 

If  ker  71  H ker  T £-  ker  S then  d(71,S,T)  = + 00 . ■ 

Proof 

Let  h£  ker  71  71  ker  T and  h £ ker  S.  Then  T(ch)  = 0,  7l(ch)  = 0 for  any 

constant  c . Then  |j  S(ch)  j|  = jc  J Jj  Sh  j|  — ■+•  00  with  |c  | — + °°.  Due  to  Lenina 

5.  1 we  get  d(71,S,T)  = + «.  ■ 

Theorem  5.1  states  that  ker  71  .0  ker  T has  to  be  contained  in  ker  S for 

d(7l,S,T)  to  be  finite.  We  prove  that  ker  71  fl  ker  T c ker  S implies  that 

the  cardinality  of  71  is  at  least  as  large  as  the  "problem  index".  Let 

ker  T - (ker  T fl  ker  S)  © A(T,S)  , 

(5.3)  * * * 

A(T,S)  = lin(?1 , ?2 , . . . , ?n*) 

where  A(T,S)  is  an  algebraic  complement  of  ker  T D ker  S in  the  space  ker  T 
and  for™  a basis  of  A(T,S)  , n = n (T,S)  s + ®. 


5.4 


De f ini t ion  5 . 1 

We  shaLl  say  Chat  index(S,T)  = dim  A(T,S)  is  the  index  of  the  problem 

k 

(S ,T) . We  shall  sometimes  write  index  (S,T)  = n . | 

Note  that  indexes, T)  = dim(ker  T)  - dim(ker  T PI  ker  S)  whenever  either 
dim(ker  T)  or  dim(ker  T P ker  S)  is  finite.  We  are  ready  to  prove  the  main 
result  of  this  section. 


Theorem  5. 


If  card(!JT)  < index(S,T)  then  d(31,S,T)  = +<*>. 


Proof  * 

n 

We  show  that  ker  31  P ker  T p ker  S.  Define  f = c T*£A(T,S). 

i=l 

We  want  to  find  a nonzero  vector  (c,,c n,...,c  *)  such  that  f£  ker  SI.  From 

1 Z n 

Lemma  4.2  it  follows  that  there  exists  an  iufoiuiatioa  operator 

t 

31  = [L  ,L  , ...,L  ] where  m = card(31)  < index(S.T)  such  that  ker  31  = ker  31. 

1 l z m ^ 


Thus  f 6 ker  31  iff  L.(f) 

j 


c.  L.(^.)  = 0 for  j = 1,2, Hence  we 
^ j *- 


get  a homogeneous  system  of  m linear  equations  in  n unknowns.  Since 

k 

m < n , there  exists  a nonzero  vector  (c,,c„,...,c  -v)  which  is  a solution 

12  n 

of  the  system.  Thus  0 * f £ A(T,S)  P ker  31,  This  means  that  a nonzero  f 
belongs  to  ker  31  P ker  T and  f ker  S.  Due  to  Theorem  5 . 1 we  get 
d(3l,S,T)  - + «.  | 

Theorem  5.2  states  that  every  information  operator  with  cardinality 
less  than  the  index  of  S and  T does  not  supply  enough  information  to  solve 
the  problem.  For  index(S,T)  * + » we  get  the  following  corollary. 


5.5 


Corollary  5.2 

If  index(S, T)  = + ® then  the  problem  (S,T)  cannot  be  solved  by  any 
information  operator  with  finite  cardinality.  | 

We  illustrate  the  above  results  for  some  restriction  operators  T.  We 
begin  with  T = 0. 

Lemma  5 . 1 (No  restriction  operator) 

Let  T = 0.  Then 

(i)  d(3l,S,0)  is  either  zero  or  infinity.  More  precisely  ker  31  'L  ker  S 

implies  d(3l,S,9)  3 + °°,  ker  31  C ker  S implies  d(3l,S,0)  = 0. 

(ii)  index(S.O)  = dim(ker  S)"^  is  finite  iff  S is  a finite  dimensional 
operator,  i.e.,  dim  S(3^)<+  ■ 

Proof 

Since  T =*  0 , ^ ^ and  ker  T = $ . If  ker  31  H ker  T = ker  31  t-  ker  S 

then  d(3l,S,0)  = + “ due  to  Theorem  5.1.  If  ker  31  C ker  S then  Sh  = 0 for 
all  h 6 V(0>  = ker  31  C ker  S.  Thus  d(31,S,0)  = 0 by  Lemma  5.2.  This 
proves  (i) . 

From  (5.3)  we  get  A(0,S)  = (ker  S)^  and  index(S,0)  = dim(ker  S)X. 
is  well  known  that  index(S,0)  is  finite  iff  S is  a finite  dimensional 
operator.  This  proves  Lemma  5.3. 

k Ik) 

As  our  second  illustration  consider  T = D , k 5 0,  i.e.,  Tf  = fv 
a scalar  function  f.  If  S is  a one-to-one  operator  then 

A(T,S)  = ker  T » {f:  f(k)  = o} 

and  index(S,D  ) ™ dim(ker  T)  * k.  Hence  we  have  to  compute  k linear  functionals 
to  assure  that  card (31)  2 index (S,Dk)  and  ker  31  fl  ker  T ■ ker  S = {o}. 


It 


for 


J 


6.1 


6.  OPTIMAL  LINEAR  INFORMATION  OPERATORS 


Assume  that  n - index(S,T)  < + °3.  We  construct  an  information  operator 

* * * 

71  with  card(7!  ) = index(S.T)  such  that  ker  ?!  P ker  T — ker  S.  Recall  that 

A(T,S)  is  defined  by  (5.3)  and  A(T,S)  = lin(|  , % , . . . , % *) . Let 

* 1 2 n 

n 

3,  = A (T,S)  •?A(T,S)'L  and  f = L*(f)§'.  + f.  where  ft  A (T , S)"*~  and 

l -■  W4  111  1 

^ 11 

L . (c  .)  = 5.  ..  Then  71  = ACT.S)^. 

i j l j 


Lemma  6 . 1 


Let 


•A.  -J-  -V  -V  ♦- 

(6.1)  7!  = [L1,L2,...,Ln*]t. 


Then  ker  7!  P ker  T C ker  S. 


Let  f 6.  ker  T.  Then  f = f + L.(f)£.  where  f £ ker  T P ker  S.  If 

0 . i i 0 

f 6.  ker  7!  then  L^(f)  = 0 for  i = 1^2^...,n  and  f = f ker  S.  This 
proves  Lemma  6.1.  ■ 

To  simplify  further  considerations  and  to  assure  that  ker  71  P ker  T c ker  S 
we  shall  consider  throughout  this  section  only  information  operators  71  such 
that  71  C 71.  (This  means  ker  71  C ker  71  and  ker  71  P ker  T c ker  71  P ker  T C ker 
due  to  Lemma  6.1.) 

We  show  the  diameter  of  d(71,S,T)  can  be  computed  in  terms  of  the  inverse 
operator  T * which  is  defined  as  follows.  (T  is  not  one-to-one  in  general.) 
Recall  that  ^ **  T(3^)  and 


(6.2)  3 = ker  T H ker  S <?  A(T,S)  0 (ker  T)X. 


6.2 


Thus  f = fQ  + fl  + f2  ^ where  fo^  ker  T ^ ker  S’  fl  ^ A(T>S)  and 

f,,  & (ker  T)^ . From  Lemma  5.1  it  follows  that  the  dependendence  of  T on  the 

diameter  d(3l,S,T)  is  only  through  the  kernel  of  31  . Let  f £.  ker  31  . Then 

f = fg  + since  f^  £a(T,S)  = (ker  31  )“*\  Define  a linear  operator  T ^ : 2^  -*  ^ 

such  that 

(6.3)  T_1z  =*  f2 


where  z = Tf. 

We  check  that  T ^ is  well-defined.  Let  z * Tf  = Tg  where 
g * gQ  + 82  ker  31*.  Since  T(f-g)  =*  0,  f-g  = (fQ-g0)  + (f2_82^  ker  T which 
yields  f9  m &2‘  Proves  that  T ^z  does  not  depend  on  a particular  choice 

of  pre- image  of  z.  Hence  T * is  well-defined. 


V 


As  An  PYpmn 1 p obs^irv^  ths t T * Q implies 
Let  K:  2^  -*  32  a linear  operator  and 
Denote 


a fnl  cT  ^ r.  n 

V, 

let  B be  a linear  subspace 


of 


(6.4) 


. | f,uP  l|Kz||. 

II  z||si,zeB 


We  are  ready  to  prove 


Lemma  6 . 2 

★ 

Let  31  C 31.  Then 


(6.5)  d(3t,SfT)  - 2||  ST_1|lr(ker  a) 


6.3 


1*1-00  f 

* - 1 

Note  that  ker  21  C ker  21  assures  that  we  can  use  T defined  by  (6.3) 

for  the  linear  subspace  T(ker  21).  Lemma  5.1  states  that  d(21,S,T)  =■  2 sup  ||  Sh  || 

where  21  h = 0 and  | j Th  { | £ 1.  Let  h =*  h^  + h^  due  to  (6.2).  Then  SI  =*  Sh0  and 

Th  = Th,, . Let  z = Th  £ T(ker  -0  . Observe  that  T ^z  = h^  and  ST  ^ z = Sh? . 

This  proves  (6.5).  I 

Lemma  6.2  states  that  the  diameter  d(2!,S,T)  is  equal  to  twice  the  norm 

of  the  linear  operator  K = ST  * in  a certain  linear  subspace  B = T(ker  21). 

This  suggests  the  following  problem.  For  a fixed  integer  n find  the  most 

relevant  information  operator  21,  card(2l)s  n,  that  is  the  operator  which 

minimizes  d(2t,S,T)  among  all  information  operators  with  cardinality  £ n. 

This  is  equivalent,  as  we  shall  prove,  to  finding  a linear  subspace  B with 

codim  B £ n-n  which  minimizes  ||  K||g  among  all  linear  subspaces  of  codimen- 
* 

sion  s n-n  . 

To  formalize  this  problem  let  ¥ be  the  class  of  all  information 

if 

operators  ^ such  that  C card(?i)  £ n where  n ^ index(S,T). 


Def  in  it  ion  6 . 1 

We  shall  say  d(n,S,T)  is  the  n- th  minimal  diameter  of  information  iff 


(6.6) 


d(n,S,T)  = h 


r 

inf  d("t,S,T)  if  n 2 index(S,T) 

26* 

n 


if  n < index(S,T) 


We  shall  say  2l°L  is  an  n- th  optimal  information  iff 


(6.7)  d(n,S,T)  =■  d(2tn’l,S,T)  . 


We  define  d(n,S,T)  = + 00  for  n < index(S,T)  since  for  any  21  with  cardinality 
less  than  index(S,T),  d(2l,S,T)  = + ®.  See  Theorem  5.2.  We  illustrate  Defini- 
tion 6.1  by  the  following  example. 


6.4 


Example  6 . 1 

Let  dim  3j  = + “ and  let  T = I be  the  identity  operator.  Define 
S = cl  for  a positive  constant  c.  Then 

(6.8)  d (n  ,c I , I)  = 2c,  vn. 

Indeed  let  11!  £ . Then  ker  HI  ^ fo}  and  d(Hl,S,T)  = 2 J ] clll^  ^ - 2c . Note 
that  -o  C’n  ( f > ) s 0 is  an  optimal  error  algorithm  since  e('j)  = c = -~d(H!,S,T)  = 

= r(Hl,S,T).  This  means  that  no  matter  how  many  linear  functionals  are  com- 
puted the  zero  of  the  space  3^  is  the  best  approximation  to  the  solution 
Sf  =*  cf  for  some  f such  that  ||  f[|  S 1.  See  Schultz  [73]  for  related  material. 

However  for  the  identity  information  operator  Ht(f)  = f we  get  ker  HI  = fo} 
and  d(I,cI,I)  = 0.  Note  that  card(I)  = + This  shows  that  d(n,S,T)  can 
be  a discontinuous  function  of  n at  infinity.  | 

From  Example  6.1  we  get  the  following  corollary. 

Corollary  6 ■ 1 

For  every  e (no  matter  how  large)  there  exists  a linear  problem  (S,T) 
with  finite  index  for  which  one  cannot  find  an  e- approximation  using  any 
finite  number  of  linear  functionals.  ■ 

We  show  that  the  n-th  minimal  diameter  and  the  n-th  optimal  information 
are  fully  determined  by  the  operator  K = ST  Let 

(6.9)  b (m, K)  - 2 inf  ||  k|L 

, codim  B ^ m 

be  the  n-th  minimal  norm  of  the  linear  operator  K. 

Suppose  there  exists  a sequence  (B^} , m 2 0,  such  that 

(6.10)  b(m,K)  = 2 j | K | Jg  and  codim  Bm  s m. 

m 


6.5 


Lot  \ = B "r  B^  and 
'JU  in  m 

k 

(6.11)  g = gn  + L (g)  Ti 

U — un  im 

i=l 


where  g.  £,  B and  = lin(Tl  ,T.  ) with  k 

Ora  m lm  2m  k,ra 


k(m) 


codim  B ^ m. 
m 


We  shall  call  B an  ra- th  minimal  subspace  of  the  linear  operator  K. 

m * 

Recall  that  L,,L_,...,L  * form  71  . See  (5.3)  and  (6.1).  Define 
1 2 n 


(6.12)  71  = [L.  ,L_  , . . . ,L  ,V,L.  *T , . . . , L,  , *.  *T]' 

n L 1 2 n”  l,n-n  ’ ’ k(n-n"),n-nw  1 


We  are  ready  to  prove  the  main  result  of  this  section. 


Theorem  6 . 1 

The  information  71  defined  by  (6.12)  is  the  n-th  optimal  information  and 
(6.13)  d(71n>S,T)  = d(n,S,T)  » b(n-n*,K)  , K = ST- 1 , n*  = index(S.T).  ■ 

Proof 

To  prove  Theorem  6.1  we  need  two  lemmas. 

Lemma  6 . 3 

Let  B be  any  linear  subspace  of  ^ with  codim  B = k < + <=.  Then  there 
exists  a unique  information  71  such  that 

(i)  51*  C 71, 

(ii)  T (ker  71)  - B, 

(iii)  card (71)  = k+n  . H 

Proof  o f Lemma  6 . 3 

= B $ B1  and  B1  - lin(Tl 
k 

have  g = gQ  + u Ll(g)T]i  where  g()  6 B and  L^T^)  = 5^. 
i*l 


,1^, . . . ,1^)  . Thus  for  every  g b ^ 


we 


Let  ^ 


Define 


6 . 6 


(6.14)  21  = [L1,C,...,L*V)L1T,L2T,...,LkTlt. 

Then  ker  21  = {f : L. (f)  = 0,L  (Tf)  = 0,  i = 1,2, ... ,n  , j = 1,2 .... ,k]  C ker  2f  . 
This  proves  (i).  Let  h G ker  71.  Then  L^(Th)  = 0 for  i = l,2,...,k  and 
Th  6.  B.  Hence  T(ker  7l>  C B.  Now  let  g be  an  arbitrary  element  of  B,  i.e., 

L.(g)  = 0 for  i = l,2,...,k.  Since  sG  1 , there  exists  f G such  that 

L ^ I 

g = Tf . Decompose  f =*  f^  + f^  + f where  f^  G ker  T ° ker  S,  f , G A(T,S) 

and  f„  G (ker  T)^"  . See  (6.2).  Then  g = Tf  = Tf,, . Since  L^(f0)  = 0 for 

i = 1,2,. . . ,n  and  L^lTf^)  = 0 for  i = 1,2, . . . ,k  we  get  f0  G ker  21  and 

g = Tfn  G T(ker  21)  . This  yields  T(ker  21)  = B which  proves  (ii). 

__  Vc  ic  *k 

To  prove  that  card (-;!)  = k+n  we  show  that  , . . . ,L  *,L^T, . . . , L^T  are 
linearly  independent.  See  Example  4.1.  Assume  that 


c.L.  + d.L.T)  f = 0,  Vf  G 3, . 
4-ir  <_  1 l.  / I 


k k k 

Set  f 88  % where  r , , . . . , ? * fom  a basis  of  A(T,S)  . See  (5.3).  Then 

L * 1 2 n 

★ k ★ k 

T?.  = 0 and  L . ( r , ) = * ..  This  vields  c.  = 0 for  i = 1,2,. ..,n  . New  let 
1 j l lj  l 

3.  = Tf.  and  set  f = f..  Since  L.CTf.)  = L.CH.)  = 5..,  we  get  d.  = 0 for 

i 1 1 j 1 J i Lj’0! 

k 

i = l,2,...,k.  This  proves  that  card  (-1)  * k+n  . 

We  now  show  the  uniqueness  of  21.  Suppose  that  an  information  operator 
2?1  ■ [L^,L? L^^*]  C satisfies  (i)-(iii).  Thus  ker  21^  C ker  21  means 

•Jf 

that  h G ker  21  ^ implies  L^(h)  =*  0 for  i = 1,2,..., n . Next  T(ker  21^)  = b 

means  that  h G ker  21^  implies  Th  G B , i.e.,  L^(Th)  = 0,  i * l,2,...,k.  Thus 

h G ker  21  and  ker  21^  C ker  21.  Since  card  (21^)  = card  (21),  from  Lemma  4 . 1 we 
get  21^'s"  21  . This  completes  the  proof  of  Lemma  6.3.  ■ 

Due  to  the  uniqueness  of  21  we  shall  write  21  = T *(B).  Note  that  21 


defined  by  (6.12)  is  equal  to  T (B  *)  where  B * is  the  (n-n  ) th 
J ^ ' n-n  n-n 


minimal  subspace  of  K. 


6.7 


Lemma  6. a 


Let  card  (Til  = n.  Then  codim  T(ker  71) 


Proof 

Let  3 = T(ker  Til  and  let  k = codim  B.  From  Lemma  6.3,'T1^  = T ^(B)  has 

•k  -k 

the  properties  71  C 71^  and  cardCTl^)  = k+n  . Repeating  a part  of  the  proof 
of  Lemma  6.3,  it  is  easy  to  show  that  ker  71  C ker  71^.  From  Lemma  4.1  we  get 

•k  •, k 

Tlj  = mTI  where  the  (k+n  )x  n matrix  M has  rank  M = k+n  . This  is  possible 
only  if  k £ n-n  which  completes  the  proof.  | 

We  proceed  to  prove  Theorem  6.1.  From  Lemma  6.3  we  know 

d (71  , S ,T)  = 2 ||  K ||  = b ('n-n* , K)  . 

n D k 

n-n 

Let  be  any  information  operator  from  Y . From  Lemma  6.4.  we  get 

n 

-k 

dim  T(ker  3»)  ^ n-n  and 

d(71,S  ,T)  = 2 1|  K|lr(ker  b(n-n*,K)  = d(3ln,S,T). 

This  proves  that  71  is  the  n- th  optimal  information  and  d(31,S,T)  = dln.S.T)  = 

•k 

b(n-n  ,K) . This  completes  the  proof  of  Theorem  6.1.  ■ 

If  d(n,S,T)  £ 2e  then  it  is  impossible  to  find  an  e- approximation  no 
matter  which  information  operator  71  with  card(7l)  £ n is  used.  In  this  case 
we  have  to  increase  n and  possibly  find  such  m > n that  d(m,S,T)  < 2e.  This 
motivates  our  interest  in  the  dependence  of  the  n- th  optimal  diameter  d(n,S,T) 
on  n.  Note  that  d(n,S,T)  is  a non increas ing  function  of  n. 

Definition  6.2 

We  shall  say  d(S,T)  is  the  diameter  of  problem  error  in  the  class  of 


information  of  finite  cardinality  iff 


6.8 


(6.  15)  d(S,T)  = l im  d(n,S,T) . 

n-« 

We  shall  say  that  the  problem  (S.T)  is  strongly  non-computable  if  d/S,T)  - + 
is  e- non- computable  if  d(S,T)  ^ 2e  > 0 and  is  convergent  if  d(S,T)  = 0.  | 

We  now  show  that  the  diameter  of  problem  error  d(S,T)  can  be  any  number. 

This  shows  that  for  any  e there  exist  linear  problems  which  are  e-non-computab le . 

Lemma  6 . 5 

Let  5 £ [0,  + <*>].  Then  there  exists  a linear  problem  (S,T)  such  that 
(6. 16)  d(S,T)  =5.  ■ 

Proof 

Let  5=4-“.  Define  T = 0 and  let  S be  a one-to-one  operator.  From 
Lemma  5.3  we  get  index(S.O)  = ■*-  « for  infinite  dimensional  . Thus  by  Theorem  5.2, 
d (n  ,S , 3)  = + ® for  any  finite  n and  d(S,T)  = + 00  = 5.  Now  let  5 £ [0,  + *). 

5 

From  Example  6.1  we  get  d(n,-^T,I)  = * for  any  n.  Thus  d(S,T)  = 5 which 
completes  the  proof.  fl 

In  the  next  section  we  show  when  the  problem  (S,T)  is  convergent  and 
how  to  find  the  n- th  minimal  information. 


7.1 


7.  CONVERGENCE  AND  MINIMAL  SUBSPACES  FOR  A HILBERT  SPACE 

In  this  section  we  assume  that  T is  a Hilbert  space.  Recall  that  the 
problem  (S,T)  is  convergent  if  d(n,S,T)  - 0 when  n — + “.  See  (6.3). 

Theorem  7 . 1 

Let  ^ be  a Hilbert  space  and  index(S,T)  < + “.  The  problem  (S,T)  is 
convergent  iff  K 3 ST  ^ is  compact.  | 

Proof 

Recall  that  K:  X -•  3o  i-s  compact  iff  K may  be  uniformly  approximated 

by  finite  dimensional  linear  operators,  i.e.,  there  exists  fK  } such  that 

n ' 

(i)  K : X - Jo  > K is  i inear , 
n °2  n 

(ii)  dim  K ( ^ ) < + 00 

(iii)  lim  ||  K - K ||  =*  0. 

n— <= 

Suppose  that  the  problem  (S,T)  is  convergent,  i.e.,  lim  d(n,S,T)  = 0. 

n~»co 

This  means  there  exists  a sequence  of  information  operators  «r5l  } such  that 

- n 

card(2l  ) ^ n and  lim  d(!H  ,S,T)  = 0.  Due  to  Lemma  6.2  and  Lemma  6.U  we  get 
n—  * 

d(2l  ,S,T)  = 2 |j  k|L  - 0,  B =*  T (ker  21  ) 
n d n n 

n 

★ • 

and  codim  Bn  £ n-n  . Let  ^ = Bn  ‘v  B^-  and  let  g = gQ  + gj  where  gQG  B , 
gjC  and  (gg.g^  - 0.  Define 

(7.1)  Kng  = Kgr 

Then  Kn  is  a linear  operator  from  3^  to  ^ and  dim  Kn(^)  = dim  K^)  4 dim  B1 
★ 

£ n-n  < + “. 


Furthermore 


7.2 


!l  «-Kn>Si|  - ||  Kg0||  * II  Kllg  ||  gg ||  - ||  Kllg  ./f|  sli2-!!  *TI|2  ‘ 

n n 1 

* IMIb  lull- 

n 

This  proves  that  jj  K-  K^j]  ^ )|  K | — 0 which  means  that  K is  compact.  This 

n 

completes  the  first  part  of  the  proof. 

Suppose  now  that  K is  compact.  Then  K may  be  uniformly  approximated 

by  [k  ].  Define  B = ker  K where  p = codim  ker  K < + «.  From  Lemma  6.3 
n p n n 

we  know  there  exists  a unique  information  operator  _■  = T (Z  ) such 

P +"  P 

“V  "k 

that  71  C 71  * and  card! 71  j.)  = P +n  . From  Lemma  6.2  we  get 

p +n"  p +n 

d (71  +n*,S,T)  = 2 ||  K,|g  . Since  Kg  = (K-K^lg  for  any  g <o  B , we  have 

p n p 

j|  Kg  ||  ^ |j  K-Kj|  j|  gj|.  Hence  ;|  Kj^  £ ||  K-K^jj  “*  0 with  n -♦  + Finally 

P 


d(P  +n  ,S,T)  ^ d (71  +n*,S,T)  s 2 1|  - 0. 

P 

Since  d(n,S,T)  is  a nonincreasing  sequence,  then  lim  d(n,S,T)  = 0 which 

n 

completes  the  Droof. 


Remark  7 . 1 

It  is  possible  to  generalize  Theorem  7.1  for  the  case  that  JV  is  not  a 

Hilbert  space  and  for  problems  (S,T)  with  nonzero  d(S,T).  We  shall  not 

pursue  the  generalization  in  this  paper.  ■ 

Theorem  7.1  states  necessary  and  sufficient  conditions  for  the  problem 

(S,T)  to  be  convergent.  Note  that  in  many  cases  K = ST  ^ is  not  compact. 

This  holds,  for  instance,  for  S = T = I and  infinite  dimensional 

We  show  how  to  find  minimal  subspaces  of  K assuming  that  K is  compact 
★ 

and  Kf^)  C 2^.  Let  K be  an  adjoint  operator  of  K.  Define  a self-adjoint 


and  compact  operator 


7.3 


<7-2'  k,  - k*k-  a.  - a, 


„ a 


Decompose  3^  = ker  ^ (ker  K^) where  the  orthogonal  complement  (ker  K)- 
is  spanned  by  eigenvectors  of  K^,  i.e.. 


(7.33  (ker  K j )“L  = lin(?1 , %2  , . . . , 


r ^ + <*> 


K % • = X.  ?. 

1 l li 


where  V > 0 and  X^  ^ ^ ...,  = S^j.  If  r is  finite  we  formally 

put  X.  = 0 and  t.  = 0 for  i ^ r+1.  Due  to  compactness  of  K,  lim  X =0. 
11  i i 

t» 

Every  element  of  f 6,  3.  has  the  unique  decomposition  f = f + (f,"  )■ 

^4  r 0 i—  “i  i 


where  fg  6.  ker  K^.  Define 


i=l 


(7.4)  Bn_n*  = ker  ^ 0 lin(§n.n*+1, , n * n , 
and  an  information  operator 

(7.3)  an(f)  - [L*(f),...,L**(f),(Tf>E1),...,(Tf,?n_n*)]t 

■k  ★ 

where  are  given  by  (6.1).  We  are  ready  to  prove 

Theorem  7.2 

•k  _ 1 

Let  3/^  f*e  a Hilbert  space,  n = index(S,T)  < + ®,  and  let  K = ST  be 
a compact  operator  such  that  K(3,  ) c • 

4 -4 

The  information  operator  51  defined  by  (7.5)  is  the  n- th  optimal  informa- 

★ 

tion,  B * defined  by  (7.4)  is  the  (n-n  ) th  minimal  subspace  of  K and 
n-n 


(7.6)  d(3ln,S,T)  - d(n,S,T)  - b(n-n  ,K)  = 2-A  * 


Proof 


7 .4 


We  first  show  that  B^  is  the  k- th  minimal  subspace,  k = n-n  . Let  f£B^. 


f = £0  + 


where  f^  £ ker  K ^ and  |J  Kf  |f  = (Kf,Kf)  = CK^f.f)  = 


i=k+ 1 


B.  K-f  1 
k 


, (f  ^ \+li'  ^ !i  • Since  this  bound  is  sharp  we  get  [j  Kj!^  = <,  >, 

i=k+l 

Let  now  B be  any  linear  subspace  such  that  codim  B ^ k.  Then  3,  = B +-  B-*"  and 

^ m 

B = 1 in(~ , , 71  , . . . , T ) where  m = codim  B s k.  Furthermore  f = f „ + L.(f)"~ 

ldm  0 .c-,  i i 

i=l 

for  certain  linear  functionals  L,  ,L_,...,L  and  f„€-B.  Thus  f£  B iff 

Id  m 0 

k+1 

L.(f)  = 0 for  i = 1,2, ...,m.  Let  f = c.f..  Then  L.(f)  = 0 for  i = 1,2,.. 

L L~ *11  1 

i=*l 

is  equivalent  to  Me  = 0 where  M = (L^(Fj))  is  the  m X (k+1)  matrix  and 

-»  t 

c = [c  ,c„ , . . . ,c^_l1  ] . Since  p < k+1  there  always  exists  a nonzero  solution 


'k+1 


k+1 


c and  therefore  a nonzero  f 


CiH  belongs  to  B.  Then 


i=l 


k+1 


Kf>? 


I - (K1f,f)  = 


hl\  a \+i II  fl" 


i=l 


which  yields  |[  K||g  2 . /\^  ^ = ||  k|[^  . This  proves  that  B,^  is  the  k- th  minimal 

* k 

subspace  and  b(n-n  ,K)  = 2»/\  • 

Note  that  B ^ = lin  (P  , , • • . , ancJ  L . ^(f ) = (f , ’L  ) for  i = 1,2,  . . . ,k. 

See  C6.ll).  Thus  the  information  operator  21^  defined  by  (7.5)  is  identical 
with  (6.12).  From  Theorem  6.1  we  get  that  31^  is  the  n- th  optimal  information 
and 

d(3tn,S,T)  = d(n,S,T)  = b(n-n*,k)  = 2-An_n*+l  • 


Then 


This  completes  the  proof. 


7.5 


Hie  information  7t  supplies  the  best  possible  information  on  the 
n 

problem  (S,T)  in  the  class  Y . Note  that  the  evaluation  of  (Tf.-J  means 
that  we  compute  the  ith  component  of  Tf  in  the  eigenvalue  decomposition  of 


8.1 


8.  OPTIMAL  ERROR  ALGORITHMS  FOR  LINEAR  CASE 


In  this  section  we  give  optimal  error  algorithms  assuming  that  S, 
and  T are  linear  operators. 


.eixa  8 . 1 


Suppose  that  for  every  f 3q  there  exists  f such  that 


(8.1)  21(f)  = 21(f)  and  T(f)  = 0. 


Then  the  algorithm  y(2t(f))  = Sf  is  an  optimal  error  algorithm  and 


(8.2)  e(cc)  = r (21,s  ,T)  = - d(21,S,T). 


Proof 


Since  h = f - f €.  ker  21  and  Jj  Th|]  £ 1,  we  get  from  (5.2) 

||  cp(2l(f ) ) - S ( f ) ||  = ||  Sh ||  * sup  ||  Sh  ||  = 4 d(21,S,T) 

hgker  21,  J|  Th  ||£1 

Thus  by  (2.13)  e(a)  = sup  ||  cp(2l(f)-S(f)  ||  £ d(2l,S,T).  However  (2.11)  and 

f 1 

(2.15)  state  that  e(-;)  2 r('Jt,S,T)  2—  d(21,S,T)  which  proves  (8.2).  ^ 

Let  21  = [Lj  ,L^ » • • • fL^]  and  card  (21)  = n.  To  minimize  combinatory 
complexity  (see  Section  9)  we  seek  optimal  (or  close  to  optimal)  error  algo- 


rithms which  are  linear,  i.e.,  "(21(f))  = L (f)g  for  some  elements 

•—  i i 


.8- 


i=  1 


r*2 sn‘ 


We  first  consider  the  case  that  k = dim  3^  < + “•  Note  that  for  every 


information  operator  21,  card  (21)  £ dim  3j  ■ 


8.2 


Leinma  3 . 2 

Let  51  = [L1,L2,...,Lk]t. 

If  card  21  = k = din  3^  < + ^ then  d(21,S,T) 

gj.g9 . • . • »gk  such  that 

k 

cp (51(f))  = L . (f ) Sg  . = Sf 

i—*  1 L 

i=i 


0 and  there  exist  elements 


is  a linear  interpolatory  optimal  error  algorithm. 


Proof 


Since  card  21  = dim  3^  < + 00 » ker  51  = [o]  and  (ker  21)"1"  = 3^ • 

5 . 1 we  get  d(5l,S,T)  = 0.  There  exist  g1>g2,...,gk  such  that  3^  = 
and  L . (g  . ) = 5 . . . Then  for  every  f 6 3,  we  get  f = L . (f ) g . . 

1 j ij  1 L L 

j_=s  \ 

that  cp(9l(f))  = Sf  is  linear  and  interpolatory.  Since  e(co)  = 0,  d 
error  algorithm. 

Without  loss  of  generality  we  shall  consider  throughout  this 
information  operators 


From  Lemma 
lin(g1 ,g2 .... ,gk) 
This  shows 
is  an  optimal 

■ 

section 


(8.3)  31  = [L1,L2 Lnj“ 

where  n = card(2l)  < dim  In  Section  4 we  showed  that  unless  ker  31  fl  ker  T 

★ 

is  contained  in  ker  S,  d(3l,s,T)  * + This  assumption  holds  if  31  C 31  where 

(8.4)  31*  = [L* ,L* L**]*  and  L *(?*)  = 5.  . 

★ 


is  defined  by  (6.1)  with  n = index(S,T)  < + 00 . Therefore  we  assume  that  21 
defined  by  (8.3)  satisfies 


★ ★ -fe 

(8.5)  L.  * L.  for  i = l,2,...,n  , n 2 n . 


J 


for  i 


8.3 


Lemma  8 . 3 


Suppose  Chat  there  exist  elements  g , >3  » • • • »gn  from  3,  such  that 


( S . 6 ^ ^ . and  TfgJ  = 0 for  i,j  * 1,2 n. 


Then  W('.'lff))  = ^ L^(f)Sg^  is  a linear  interpolator'/  optimal  error  algorithm 


and 


i=l 


e(cc)  = r(21,S,T)  = ~ d(-(,S,T)  . 


Proof 

n n 

Let  f = L.(f)g..  Then  L.ff)  = ) L.(f)L.(g.)  = L.(f)  for  i = 1,2,.  . . ,n. 

/ , J J i j i aj  i 

j=i  j=i 

Thus  -t ( f > = '.'Iff)  and  cpf'-’lff))  = Sf  which  proves  that  co  is  a linear  and  inter- 
polator/ algorithm.  Since  T(f)  = 0,  Lemma  8.3  follows  from  Lemma  8.1.  ■ 

Lemma  8.1  and  3.3  are  not  applicable  unless  there  exists  an  element  f 
which  shares  the  same  information  as  f and  belongs  to  ker  T.  We  now  show  how 
to  find  a linear  optimal  (or  close  to  optimal)  error  algorithm  in  general. 

Let 


(8.7)  = T(ker  Jl)  'V  Tfker  -ff*' 

where  Tfker  -0  = lin(T|^  , ~lt> , . . . 1 1^)  . From  Lemma  6.4  we  know  that 

X * 

k = dim  Tfker  .1)  ^ n-n  . Then  for  every  g £ 3,  we  have 

k 

(8.8)  g = gQ  + L (g)  \ 

i=l 


where  gQ  £ Tfker  21)  and  RJ(RV,. 
tionals  such  that  R (TJ.)  = 6 

1 J ij 


. , are  linearly  independent  linear  func- 
Def  ine 


(8.9)  c “ sup 

g€^ 


II  SQ  II 


8.4 


Note  that  c depends  on  ker  21  and  T but  is  independent  of  3^  and  S.  Further- 
more c ^ 1 and  if  T(ker  -0  is  closed  then  c is  finite.  If  X is  a Hilbert 

space  then  we  can  assume  that  T(ker  21)'*’  is  the  orthogonal  complement  of 

k. 

T(ker3T>,  = 5^ , R^g)  - (g.'iJ  and  }j  § J/2  = jjg0|f  + ^ |(g,T)|2 

which  implies  c = 1 . 

n 

Let  3,  = ker  31  9 (ker  21)  . Then  f = f + L . (f)  5 . where  f.  ker  21 
I 0 t—  i "i  0 


and 


Li(V  = “i j for  = 1>2’---’n> 


* ^ * * 
for  i = l,2,...,n  , where  F . 


is  defined  by  (8.4).  Note  that  Tf  = Tf„  + L.(f)F.  since  TF*  = 0. 

0 V i 1 L 

Thus  i=n  +1 

k — d im  1 in  ( ^ > • • • > T 


k k k 

There  exist  linearlv  independent  elements  , such  that  Tt  = 0 

~n  +1  n-k  ~l 


and  £ lin(?n*+1 ^) , i = n +1 , . . . ,n-k . 


-u  J- 

Let  m = n-k.  Since  X = T(3,)  there  exist  F , ...,F  such  that 

a i tIH- 1 "n 


' . = T F . for  i = l,2,...,k.  Define 
1 m+-  l 


(8.10)  M = Lt(§  ))  i,J  = 1,2,. ...n. 


We  show  that  M is  nonsingular.  Indeed,  let 


ClLi(  = l)  + + CnLi(0  = 0 for  1 3 1.2 

k -k  .u 

Then  - = c.F  + . . . + c p £ ker  21  and  TF  £ T(ker  21).  Since  Tp.'  * 0 for 
11  n n i 

k k 

i • 1,2,.. . ,m«n-k,  we  get  T*  = c .T  * . + ...  + c T5.  ■ c L . T|  + ... 

rrH-1  riH-1  n n m+1  1 

4*  c Tl  £ T(ker  'Jt)  . This  implies  c , 3 . . . 3 c 53  0.  Hence 

n k m+1  n 

* * ^ x 

§=cF  4-,..+c§  ^ (ker  21)  which  yields  c = ...  = c * 0.  This  proves 
11  mm  1 m r 

that  M is  nonsingular.  Define 


' J 


8.5 


t t - 1 _*  * t 

(8.11'  S j »80  » • • ■ » 8n  = ( '1  * [ j j i 


Note  that  L.(?,)  = 6..  for  i,i  = 1,2 m.  We  are  ready  to  prove  the  main 

i J ij 

result  of  this  section. 


Theorem  8.1 

Let  n = index(S,T)  = card('-l)  < + 
by  (8.9)  and  (8.11)  respectively.  Then 


re.  Let  c,g  , ^.....g  be  defined 


(3.12)  9(31(0) 


L (f)Sg 


i*l 


is  linear,  interpolatory  and  an  optimal  error  algorithm  within  a factor  of 
c,  i.e. , 


(3.13)  r(3l,S,T)  ^e(rp)  £-d(3t,S,T)  r(3l,S,T). 


Proof 


Since  Tf  = Tf  + , L.(f)T|.  we  get 

0 i_  i t ° 

i=l 


(3.14)  R.(Tf)  = L.(f)  R . (T? . ) for  j 

j Lu  i j i 

i-1 


= 1 9 h 


Je  ★ 

Set  f = £ f°r  f = 1,2 n in  (8. 14).  Since  = 0 for  i £ n-k  and 

i k 

R . (T?  . ) = R.(T).  ) = 6.  . for  i > in,  we  get 

J -i  j'  i-m  i-m,j 


(8.15)  [0,1]  - MM 


where  0 is  the  k x (n-k)  zero  matrix,  I is  the  k X k unit  matrix  and 


Mj  - (Ri(TSj))  i - 1,2, 


k,  j - 1,2 


8.6 


Since  M is  nonsingular  we  have 


- 1 


(8.16)  Mj  = [ 0 , I ]M  . 


From  (8.11)  and  (8.16)  we  get 


(8.17)  Tg.  = R.(T5.r  .. 

1 j 1 j 


J-l 


We  are  ready  to  prove  optimality  of  cp(2l(f)) 
n 


L . (f) S«  . . Let 

t-*  l - i 

i=l 


f = L . (f)g  . for  f G 3n  • Since  L.(g.)  = 5..,  then  21(f)  = 31(f).  Thus 

t—t  1 L 1 J lj 

i=»l 

cp(2l(f))  = Sf  is  an  interpolatory  algorithm.  Let  h = f-f,  h £ ker  31.  From 
(8.17),  (8.14)  and  (8.8)  we  get 


Th  = Tf  - / L . (f ) Tg  . = Tf  - L.(f)  R.(T?.)T(.  = 

L*  1 t 


i=l 
k , n 


i-1  J-l 


Then  Th ' 


= Tf  - / / L. (f)R.(Te.)  T).  = Tf  - R.(Tf)T). 

L\  L 1 j v J “ J J 

j-i\i-i  ' J-l 


(Tf)  € T(ker  31)  . 


(Tf)  ||  £ c ||  Tf  ||  £ c,  due  to  (3.9)  . Thus 


||  <p(31(f))  - Sf  ||  = ||  Sh  ||  * c sup  ||  Sh  ||  d(2l,S,T)  £ cr(31,S,T). 

h€ker  21 

II  Th||si 

Q 

Since  f is  an  arbitrary  element  of  3q>  e(cp)  d(21,S,T)  and  of  course 
e(rV)  2 r(21,S,T).  Hence  (8.13)  is  proven  which  completes  the  proof. 


3.7 


i 

Remark  3 . 1 

Theorem  2.2  assures  us  of  the  existence  of  an  interp  ilatory  algoritlm 
whose  error  is  within  at  most  a factor  of  two  of  the  optimal  error.  Theorem 
3.1  shows  how  to  construct  a linear  interpolator^/  algorithm  whose  error  is 
within  at  most  a factor  of  c of  the  optimal  error.  | 

Note  that  if  c = 1 then  cp  defined  by  (8. 12)  is  optimal.  This  leads  to 

Coro! larv  3 ■ 1 

If  1 is  a Hilbert  space  then  the  algorithm  rp  defined  by  (3.12)  is  a 
linear  interpo latory  optimal  error  algorithm  and 

e(®)  - r(5t,S,T)  = ||  ST-1^^  ^ > ■ 

l 

t 

Corollary  8.1  and  Lemma  5.1  yield 
Corollary  3.2 

I If  ^ is  a Hilbert  space  then 

d(3t,S,T)  = 2r(3t,S,T)  . ■ 

The  problem  of  linear  optimal  error  algorithms  was  considered  by  many 

authors  including  Bakhvalov  [71],  Bojanov  [74]  and  [76],  Micchelli  and 

f 

Rivlin  [77]  and  Smolyak  [65].  They  assumed  that  the  solution  operator  S is 
a linear  functional  and  3q  is  a convex  and  balanced  set.  See  also  Golomb 
and  Weinberger  [59]  for  some  related  material  and  discussion. 


* 


9.1 


9.  COMPLEXITY  FOR  THE  L INEAR  CASE 

We  specify  our  model  of  computation  for  linear  problems  (S,T)  and 
linear  information  operators  21  as  follows.  (For  the  general  case,  see 
Section  3.) 

Model  of  Computation  for  the  Linear  Case 

(i)  Let  P be  a given  collection  of  primitives.  We  assume  that  the 

addition  of  two  elements  of  32 > f+g > and  the  multiplication  of  an 
element  of  ^ by  a scalar,  cf,  are  primitive  operations  which 
belong  to  P.  We  also  assume  that  every  linear  functional  L, 

L:  3^  "*  (L  > is  a primitive  operation  which  belongs  to  p.  This 
implies  that  any  linear  information  operator  31  = [ , L^ , . . . ] 
of  finite  cardinality  is  permiss  ible  where  L^,L^,...,L  are 
arbitrary  linear  functionals. 

(ii)  To  normalize  the  measure  of  the  complexity  we  assume  that  the  cost 
of  the  addition  of  two  elements  of  3^  an<i  the  multiplication  of  an 
element  of  3?  by  a scalar  is  taken  as  unity.  Note  that  for  a 
finite  dimensional  space  32 » m = dim  32  > untt  cost  means  the  cost 
of  m scalar  additions  or  multiplications. 

Let  comp(L)  denote  the  complexity  of  evaluating  a linear 
functional  L.  Let  21  = [L^,L9 , . . . .L^]  be  a linear  information 
operator  with  linearly  independently  linear  functionals  L ^ , L,, , . . . , , 

card(21)  = n.  We  assume  that  21(f)  is  computed  by  the  independent 
evaluation  of  L^(f) ,L^ (f) , . . . ,Ln(f)  and  the  information  complexity, 
see  (3.4),  of  21  is  given  by 


9.2 


fW 


n 

(9.D  comp  (21)  = comp(L  ). 

L 

i=  1 

If  comp(L^)  = Cj  Chen  comp(21)  = nc j which  shows  how  the  informa- 
tion complexity  depends  on  the  cardinality  of  21. 

(iii)  Let  » be  a permissible  algorithm  which  uses  21(f)  and  finds  an 
e-approximat ion  to  or  = S(f).  Let  d(rp)  be  the  combinatory  com- 
plexity of  cc.  For  all  problems  of  practical  interest,  p has  to 
use  every  L^(f),  i ~ l,2,...,n,  at  least  once  and  d(-p)  ^ n- 1 . 

We  rule  out  special  problems  and  information  operators,  assuming 
that  d('p)  5 n- 1 for  every  algorithm  under  consideration.  B 

Example  9 . 1 

Let  P = {arithmetic  operations,  the  evaluation  of  linear  functionals, 
the  evaluation  of  a function,  the  evaluation  of  a derivative}.  Let 
21(f)  = {f(x),f’(x)}  where  f is  a func  tion  in  a space  of  dimension  m,  m £ 
Assume  ?;(2Uf))  = x - (f'(x))  ^f(x). 

Case  1.  m < <=.  Let  comp(L)  = c^,  for  every  linear  functional  L.  Then 

2 

conp(f(x))  = mc^,  comp(f'(x))  = m c^.  We  compute  rp(21(f))  by  solving  the 

appropriate  linear  system  by  Gaussian  elimination.  Thus,  by  (ii), 

2 

d(u)  = 0(m  ) times  unit  cost.  We  conclude 

2 

comp(cp)  = me  ^ + itTcj  + d(cp). 

Case  2.  m = «.  We  add  the  solution  of  a linear  system  to  our  set  of  primi- 
tives; let  the  complexity  of  this  primitive  be  c^ . Let  comp(f(x))  = cn, 

comp(f'(x))  * C-.  Then  comp(cp)  • c„  + c,  + c.  . ■ 

j 2 j A 


4. 


9.3 


This  model  of  computation  is  an  idealized  one  since  we  assume  that 

every  linear  functional  is  a primitive  operation.  However,  even  in  this 

idealized  model  we  shall  prove  that  the  complexity  of  a linear  problem  (S,T) 

as  a function  of  e can  be  essentially  any  decreasing  function  of  e. 

n 

Let  <p  be  a linear  algorithm,  i.e.  co(31(f))  = : L,(f)g.  for  certain 

l i 

i=l 

elements  g^,g?,...,gn  from  ^ • The  element  g^  depends  on  71,  S and  T but  is 
independent  of  f.  Therefore  the  elements  g,»g_,...,g  can  be  precomputed. 
The  computation  of  cp(71  ( f ) ) , given  7l(  f ) , requires  at  most  n multiplications 
and  n- 1 additions  which  are  primitive  operations  with  unit  cost.  Thus,  any 
linear  algorithm  o is  permissible  and  its  combinatory  complexity  is  at  most 
2n- 1 . Due  to  (iii),  every  linear  algorithm  is  within  a factor  of  2 of 
minimal  combinatory  complexity.  Therefore  to  make  combinatory  complexity 
small  it  is  desirable  to  make  the  cardinality  of  31  as  small  as  possible 
under  the  constraint  that  r(3l,S,T)  < e. 

Fix  31,  S and  T.  Recall  that  $(e)  is  the  class  of  all  permissible  algo- 
rithms cp  which  use  the  information  operator  31  and  whose  error  is  not  larger 
than  e,  e (cp)  £ e. 

Lemma  9 . 1 

Let  r(31,S,T)  < e.  Assume  there  exists  a linear  algorithm  cp  in  5(c). 

Then, 


(9.2) 


9.4 


Proo  f 

Since  a linear  algorithm  is  permissible,  $(e)  is  not  empty.  The 
complexity  of  a linear  algorithm  cp,  comp(rp)  , is  not  larger  than  comp(21) 
Since  comp(31,  S ,T,  e)  2 comp(2l)  + n- 1 , (9.2)  is  proven. 


2n-  1 , 


Note  that  if  comp(21)  s n then  (9.2)  yields 

(9.3)  i 1 ^ comp  (31,  S ,T , el  ^ ^ 

' 2 3n- 1 comp(cp) 

In  many  cases  comp  (31)  » n which  yields 

(9.4)  comp  (31,  S ,T , e)  — comp(rp)  - comp(3t) 
for  every  linear  algorithm  rp. 

Fix  S,T,  and  let  e tend  to  zero.  We  seek  the  minimal  cardinality  of  an 
information  operator  31  such  that  r(31,S,T)  < e.  Let  Y be  a class  of  permissible 
information  operators  such  that  inf  r(2t,S)  = 0.  See  (3.3).  Define 

3£v 

(9.5)  m(Y,S,T,c)  = min  [card  (21)  : 31  6y,  r(21,S,T)  < e}. 


We  shall  say  m(Y,S,T,e)  is  the  e-cardinal itv  number  for  the  problem  (S,T) 
lr.  the  class  Y.  Note  that  m(Y,S,T,e)  is  a nonincreasing  function  of  e. 

Let  Y be  the  class  of  all  linear  information  operators  21  such  that 

CO 

card(?J)  < + ®.  (Ut  is  permissible  due  to  (i)f.)  Note  that  U C ¥ where 

* n U 

n=n 

Y , see  Section  6,  is  the  class  of  all  information  operators  ^ such  that 

ao 

31  c 51,  card(21)  S n.  Furthermore  U Y^  ■=  Y if  index(S,T)  = 0. 

n=0 

The  class  contains  all  information  operators  of  practical  interest 
since  every  information  operator  which  is  to  be  computed  has  to  have 
finite  cardinality.  Therefore  the  e-complexity  comp^^.S.T,  e)  is  the 
inherent  complexity  of  the  problem  (S.T). 


9.5 


We  show  that  m(V^,S,T,e')  can  be  essentially  any  decreasing  function  of 
£.  More  precisely  assume  that  e belongs  to  the  interval  (O.e^].  Let 

(9.6)  g:  (Q,c0]  - R 

be  a decreasing  function  such  that  g(eQ)  ^ 1 and  lim  g(e)  =■+“>. 

c-0+ 

Theorem  9 ■ 1 

For  every  function  g defined  by  (9.6)  there  exists  a linear  problem 
(S.T)such  that 

(9.7)  g(c)-l  < m(YufS,T,e)  £ g(e)  , Vt  €(O,«0]. 

Furthermore  there  exists  a sequence  [e.}  such  that  e . (0,€g],  lim  = 0 

1 i-*“ 

and 

(9.8)  m(Yu,S,T,e.)  = g(e.) . ■ 

Proof 

Let  g"1;  [g(«(J),  + °°)  - R+  be  the  inverse  function  of  g.  Define 

B.  » e +1  for  i < g(e  ) and  0.  - g_1(i)  for  i 2 g(e  ).  Note  that  lim  0.  = 0. 
1 U U 1 *CD 

Let  3^  = ^ =•  lin(§^,§2»*  • •)  be  an  infinite  dimensional  Hilbert  space 
where  (?.,?.)  =*  5...  Define  T - I and 

i J ij 

CD 

(9.9)  Sf  - ^ ei(f,?i)§i  . 

i“l 

Thus  S is  a self-adjoint  and  compact  operator.  Furthermore  Sfi  = 0^^  for 

i » 1,2 

Note  that  n =0  where  n “ index (S, I).  From  (7.2)  we  get  3 S and 
the  eigenvalues  of  satisfy  X^  3 for  i 3 1,2,...  • 

| * * “ 1 “ r * 


9.6 


Let  m = m(Y  ,S,I,e).  This  means  there  exists  an  information  operator  21^ 
such  that  card(2t^)  = m and  r(2l^,S,T)  < e.  Moreover  for  every  21  such  that 
card(2t)  < m,  r(21,S,T)  ^ e.  Due  to  Theorem  7.2  and  Corollary  8.1  we  know  that 

Swl  5 r<VS>T>  < *• 

Thus  m+1  2 g(e  ) + 1 and  3 , = g ^(dh-1)  < e which  yields  m > g(e)  - 1. 

U m+- 1 

Furthermore  for  21  = 21  defined  by  (7.5)  we  get  card(21  ) < m and 

m-  1 n>-  i 

r(21  _,S,I)  = 3 ^ e.  This  yields  m £ g(e)  and  proves  (9.7). 

nr  1 m 

Let  “ 3^+^  for  i ^ g(Cg).  Then  6 (0,£q]  and  lim  = 0.  Since 

i 

r(2li>S,I)  = = «i  we  get  mfY^.S, I, e>  = i+l  = g(e;).  This  proves  (9.8) 

and  completes  the  proof.  ■ 

Theorem  9.1  states  that  m(Y^,S,T,e)  can  be  an  essentially  arbitrary 
function  of  e.  Recall  that  the  e-complexity  comp (Y  , S ,T , e)  of  the  problem 
(S,T)  in  the  class  is  defined  by  (3.9).  From  Theorem  9.1  we  can  conclude 
that  comp(Y^,S,T, e)  can  depend  arbitrarily  on  e.  To  show  this,  assume 
for  the  sake  of  simplicity  that  the  complexity  of  evaluating  any  linear 
functional  is  fixed,  comp(L^)  = c^. 

Theorem  9.2 

For  every  function  g defined  by  (9.6)  there  exists  a linear  problem 
S,T  such  that 

(9.10)  g(c)(Cl+l)  - cx  - 2 < comP(Yu,S,T,e)  s g(c)(Cl  + 2)  - 1,  Ve  e(0,e0).  ■ 

Proof 

Consider  the  problem  S,T  defined  in  the  proof  of  Theorem  9.1.  Thus, 
the  c-card inal ity  number  m = m(  Y^,S,T,e)  satisfies  (9.7)  and  the  information 
complexity  21  such  that  card(2l)  * m,  r(2l,S,T)  < e,  satisfies 


9.7 


(9.11)  (g  ( e)  - DCj  < comp(21)  = m( , S ,T , e) c j ig(c)cJ. 

Since  the  problem  (S,T)  is  defined  in  a Hilbert  space,  Corollary  8.1  assures 
the  existence  of  a linear  optimal  error  algorithm  t,  e(rs)  =*  r(21,S,T)  < e. 

Thus 

(9.12)  m-  1 + comp  (71)  £ comp  (7  , S ,T , e)  £ comp  v £ 2m- 1 + comp  (71)  . 

From  (9.11)  and  (9.12)  we  get  (9.10).  ■ 

Theorem  9.2  states  that  comp (Y , S , T, e)  is  roughly  the  same  function  of 
e as  the  e-cardinality  number  m(Y^ , S ,T , e)  . Note  that  the  function  g can 
tend  to  infinity  arbitrarily  fast  as  e tends  to  zero.  This  proves 

Corollary  9 . 1 

(i)  There  exist  linear  problems  with  arbitrarily  hard  complexity. 

(ii)  There  are  no  "gaps"  in  the  complexity  functions.  | 

This  may  be  contrasted  with  the  theory  of  recursively  computable  func- 
tions where  complexity  gaps  are  known  to  occur  (Borodin  [72]). 

Remark  9 . 1 

We  assumed  that  21  consists  of  linear  functionals  which  are  computed 

independently  and  therefore  comp(2l)  **  nc  ^ . For  some  information  operators, 

71(f)  can  be  computed  faster  than  nc^.  For  instance  assume  that  L^(f)  = f(x^) 

for  distinct  points  x^,  i = l,...,n,  where  f is  a polynomial  of  degree  n-1. 

2 

Then  the  complexity  of  is  0(n)  but  71(f)  can  be  computed  in  0(n  log  n) . 

In  fact.  Theorem  9.2  remains  valid  under  the  relaxed  assumption  that 

comp(7l(f))  “ <®(n)  where  jj  is  an  increasing  function  cf  n with  1 im  i»(n)  m -t-  ®.  ® 

n-*® 


/ 


10.1 


CHAPTER  111 
APPLICATIONS 


In  this  section  we  apply  the  general  theory  to  a variety  of  problems. 
We  consider  rather  simple  problems  since  we  wish  to  show  the  importance  and 
usefulness  of  the  ideas  and  avoid  overwhelming  the  reader  with  technical 
details . 

10.  APPROXIMATION  IN  A HILBERT  SPACE 


Let  H = lin(T  , r ,...)  be  an  infinite  dimensional  Hilbert  space  over 

' 9 2 QD 


the  real  field  R 


where  (f.  ,§.)  = 5 ..  Thus  f H iff  f 
1 J ij 


(f,a.)".  and 
i i 


i=l 


(f,?.)  < «.  Let  {S^}  be  a nonzero  sequence  of  real  numbers  such  that 


i=*l 


1 3 I £ I 9.  ,1  for  all  i.  Let 
1 i1  1 l+l1 

00 

(lo.i)  = ^2  3 (f:  f £ H and  u < + “}• 

i=l 


We  consider  an  approximation  problem  defined  by 


(10.2)  Sf  ■ f,  f € 30 

where  3Q  - [f:  f € ^ and  ||  Tf||  £ l)  for 
00 

(10.3)  Tf  - ^ 9t<f*W  VT{3!)CHl 

i-1 

We  first  find  the  index  n = index(I,T).  Since  ker  S = [0],  (5.3) 

★ 

yields  A(T,S)  3 ker  T and  n = dim(ker  T) . Let  i^  be  the  largest  index  such 

; * 

that  g =*  0 for  i = l,2,...,i0  (g£  +1  f 0)  . Then  n = iQ<  + ®. 

71*(f  i =•  [(f  ,'  ) ,(f  ,s2) (f.?n*)l^  satisfies  Lemma  6.1,  i.e., 


Note  that 


10.2 


ker  71  T = [0]  - ker  S.  From  (6.3)  we  easily  find  the  inverse  of  T , 


T'l£.  ' 

. V A 

i=n  +1 

- 1 - 1 1 
and  K = ST  = T . Mote  that  K is  self  adjoint  and  K".  = — r . Thus  K is 

‘i  S . i 
i 

contact  iff  1 im  5^  = + <=.  From  Theorem  7.1  it  follows  that  the  problem  (I,T) 
is  convergent  iff  lim  5.  =»  + “. 

We  want  to  find  the  n-th  optimal  information  71  for  n • n . Since 


^ - 1,  > we  get  from  (7.5)  that 


(10.4)  3tn(f)  = [(f,'1),(f,!2),...,(f,Sn)]t 


is  the  n-th  optimal  information  and  due  to  Theorem  7.2  and  Corollary  8.1 
we  have 

(10.5)  r (71  ,S,T)  = - l-  i . 

n s . 

1 n+ 1 1 

The  linear  optimal  error  algorithm  ro  defined  in  Theorem  8.1  is  given  by 
n 

(10.6)  (f))  = . (f,C.)?.,  e(«p)  = 

n i l 

i=l 

Note  that  cp  is  the  initial  section  of  f = ^ (f,§^)^. 

i=l 

We  analyze  the  complexity  of  this  problem.  Recall  that  Y is  the  class 

of  all  linear  information  operators  with  finite  cardinality.  It  is  easy  to 

see  that  the  e-cardinality  number  m = m(Y^, I ,T, e) , see  (9.5),  is  equal  to 

] 

the  smallest  number  n such  that  jr^ — r < e.  Since  there  exists  a linear 

• °n+ 1 1 

optimal  error  algorithm  for  any  e we  get  bounds  on  the  e-complexity  of  the 
problem  (S,T)  in  the  class  Y , 


10.3 


(10.7)  (c1+l)m(*u,I,T,e)  - 1 £ comp  ( ^ , I ,T,  e)  < (c^lmC^,  I,T,e)  - 1 

where  is  the  complexity  of  evaluating  a linear  functional  of  the  form 
(f»*j_)»  Note  that  m depends  only  on  how  fast  3^  goes  to  infinity  and  due 
to  Theorem  9.1,  m = mCf^ , I ,T , e)  can  be  essentially  any  decreasing  function 
of  e. 


11.  UNIFORM  APPROXIMATION 


Let  k be  an  nonnegative  integer.  Let  3^  = C[0,1]  for  k = 0 and 

3 =■  if:  f6  Ck-1[0,l],  f(k)£  L r 0 , 1 ] } With  ||f||  = max  |f(t)|.  Let 

1 OSt^l 

3o  “ and  \ = LJ0.1],  Define 

(11.1)  Sf  = f , Tf  = — f (R) • 

K.  • 

(k)  _ i * 

Since  ker  T = ‘f:  f = 0],  we  obtain  n = dim(ker  T)  = k.  Consider  an 
information  operator 

(11.2)  21(f)  = [f(t1),f(t^),...>f(tn)]t 

for  d is  tine  t t^  6.  [ 0 , 1 ] , i = 1,2, ...n  with  n sk.  Note  that  ker  21  ~ ker  T 3 
‘.0 } = ker  S . 

For  k = 0,  Example  6.1  yields  r (21, 1,1)  = 1 for  any  information  operator 
with  finite  cardinality.  To  assure  convergence  of  the  problem  (I,T)  we 
assume  k £ 1.  We  consider  two  cases. 


Case  1 , n 


Let  .u(t)  = ( J ( t- 1 ) and  g.(t)  = <ju(t)/ {(t- 1 . ) uu’  (t . ) } for  i = 1,2,  ...,n. 
i=  1 1 1 11 

Observe  that  g^(t^)  3 6^j  and  = 0.  From  Lemma  8 . 3 we  know  that 


n 

(11.3)  cp(21(f))  = f(t  )g  (t) 

L-t  11 

i=l 

is  a linear  interpo latory  optimal  error  algorithm.  Furthermore  Sf(t)  - cp(21(f)) 
g(t)jj(t)  where  g(t)  = f (t  , t^  , . . . , t^;  t)  is  the  n-th  divided  difference  of 
f and  ||  gjj  ^ 1.  This  yields 


11.2 


(11.4)  e(cp)  = r(2l,  I,T)  =■  ||  u)||  . 

Jr 

We  seek  optimal  points  t.,...,t  which  minimize  e('j)  = e (">,  t.  , t.  , . . . , t ). 

in  1 2 n 

•k  -fc 

It  is  well-known  that  are  uniquely  defined  by  the  zeros  of  the 

1 -fr  tt  i- 1 

Chebyshev  polynomial  T (-r(t+l)),  i.e.,  t.  = 2 cos(-—  + n)  - 1 for 

n Z l 2n  n 

i = 1 ,2  .... ,n  and 


(11.5)  e(-s,t*,...,t*)  = . 

4 


Case  2 . n > k . 


Micchelli,  Rivlin  and  Winograd  [76]  show  there  exists  a linear  optimal 
error  algorithm  cp  such  that  e(">)  = r(2l,  I,T)  and 


(11.6)  r(21, 1,T)  = ||  q 


where  q is  a perfect  spline  of  degree  k having  at  most  n-k  knots  and  q(t  ) = 0 

(k)  t 

for  i ■ 1,2,..., n,  II  — II  = !•  Furthermore  if  t^  = 0,  t = 1 and 

A 3 max  (t.  ,-t.)  they  prove 
n ....  , l+l  i ' 

l^i ^n- 1 

.k 

k’  k 

(11.7)  s r(3l,IfT)  V 

4 4 

★ ★ 'k 

Furthermore  the  points  > fc2 * * ‘ » tn  which  minimize  r(2l,  I,T)  = 
r(2l, I,T;  t^,t0 , . . . .t^)  are  the  zeros  of  the  Chebyshev  perfect  spline  of 
degree  k with  n-k  knots  and  r(2l,I,T;  t^,...,tQ)  is  the  n-width  of  In  the 
sense  of  Gelfand  (Tichomirov  [69])  and 


(11.8)  r (21, 1 ,T,  t , t , . . . , t ) * c hk,  V n > k, 
i z n k y n 


where  0 < c.  £ c.  £ ct  < + ® for  some  constants  c.  and  c.  , h ° 

— k k,n  k — k k n 


11.3 


We  analyze  Che  complexity  of  uniform  approximation.  Let  Y be  the 

class  of  a 1 1 information  operators  It  such  that  21(f)  = [ f (t  , f (t^ f (t  ) J1” 

2 

for  any  distinct  t.  and  any  n.  If  —r  < e then  choose  n = k and  (11.5) 

1 4 

assures  that  the  e-complexity  of  the  problem  (I,T)  in  the  class  Y satisfies 


(11.9)  (Cj+l)k-l  s comp  (Y,  I ,T,  e)  ^ (c^+2)*k-l 


where  is  the  complexity  of  one  function  evaluation, 
n = n(e)  be  the  smallest  integer  such  that 


let 


(11.10)  n > 


k.n 


iv— 

• /e 


Due  to  (11.8)  we  get  < n(e)  £ i^T/.^e  + 1.  We  have 

tC  K 

(11.11)  (Cj  + l)n(e)-l  £ comp(Y,  I,T,e)  ^ (c^-r-2)n(e)  - 1. 


Note  that  (11.11)  is  tight  for  c^  » 1.  The  conditions  (11.10)  and  (11.11) 
state  that 

comp(Y, I,T, e)  = 

This  shows  that  asymptotically  in  e,  complexity  decreases  as  the  regularity 
of  the  class  of  problem  elements  increases. 


12.1 


12.  EVALUATION  OF  INTEGRALS 


Let  k be  a nonnegative  integer.  For  k = 0 let  3^  = C[0,1]  and  for  k s 1 
let  3j  = U : £<k)  piecewise  continuous  on  [-1,1]}.  Define  32  = 

3,  =*  L_[-l,l]  and 

*4 

(12.1)  Sf  =*  ' f(t)dt  and  Tf  = ~r  f(k). 

v k * 

-1 


Consider  the  information  operator  21  given  by 


(12.2)  21(f)  = rf(t  ),...,f(j_1)(t  ),...,f(t m),...,f(j'1)(t  )]C 

i i m m 


(j-1) 


for  -1  £ tl  < t0 


t £ 1 where  j = 1 for  k = 0 and  j-1  < k for  k 2 1, 
m J 


Note  that  card(2l)  = j m.  Since  n = diam  A(T,S)  = k - f— ^—1  we  have  to  assume 


I m S k 


~k-  r 

L 


I 2 


For  k = 0 it  is  easy  to  verify  that  r(2t,S,I)  = «, fi  no  matter  how  many 
function  evaluations  are  known.  To  assure  convergence  of  the  problem  (S,T) 
we  have  to  assume  k ^ 1.  For  k = j this  problem  was  considered  by  3ojanov[76] 
who  showed  the  existence  of  a linear  optimal  error  algorithm  'p.  His  analysis 
is  based  on  Smolyak's  lemma  (Smolyak  [65],  Bakhvalov  [71]).  Note  that  Theorem 
8.1  and  Corollary  8.1  assure  the  existence  of  a linear  optimal  error  algo- 
rithm defined  by  (8.12)  for  any  j and  k. 

For  the  sake  of  simplicity  we  assume  here  that  k = j.  Then  the  error 
e('p)  = r(21,S,T)  is  given  by  Bojanov, 


(12.3)  r(2l,S,T) 


ft  -l)^k"^  f 1- 1 \2k+l  m~  ^ /t  -t 

(ti  I}  (1  v 2 * ri+i  ti 

2k+l  2k+ 1 k 

i-1 


2k+l  ^ 


where  E^  is  the  minimal  error  of  approximation  of  the  function  t by 


12.2 


polynomials  of  lower  degree  in  the  space  I.  . From  Akhieser  [56,  p.  191  we 


get 


(12.4)  E, 
k 


(k1.)2 

\2k+l  (2k*. ) 


Jt;  (1  + o(k) ) . 

2 4k 


How  should  the  points  t,,t  , ...,t  be  chosen  so  as  to  minimize  the  radius 

1 z m 

r(j»,S,T)?  Bojanov  shows 


(12.5)  min  r ('Jl.S.T,  t,  , . . . ,t  ) = h E, 

1 m m k 

i 


where  h 

m 

equal  to 


(m- 


, 2u/~2k+l 

1 -r  v — ^ — 


(1  + o(m))  and  the  optimal  points  are 


(12.6)  t*  = -1  + (E?(2k+l)/2) 1 (2k)  • h + 2 (i- l)h  . 

l K mm 

* * 

Note  that  t^,...,t  are  equally  spaced. 

Observe  that  commonly  used  Gauss  quadrature  is  not  an  optimal  error  algo- 
rithm since  it  is  based  on  the  zeros  of  orthogonal  polynomials  which  are  not 
equally  spaced. 

We  analyze  the  complexity  of  evaluation  of  integrals.  Let  f be  the 
class  of  all  information  operators  of  the  form  (12.2)  for  any  distinct  t^, 
any  m,  and  with  k = j.  Let  m = m(e)  be  the  smallest  integer  such  that 

(12.7)  hk  E.  < e. 

m k 

For  small  e,  m = — rx(  1 + °(D)*  From  (12.7)  we  get  bounds  on  the  e-complexity 

4c 

comp  (Y, S ,T , e)  , 

k k 

(12.8)  m(e)  u c(f(i_1))  + km(e)-l  S comp(Y,S,T,  e)  * m(e)  ^c(f(l'1})  + 2km(e)-l 

i-1  ia 1 


12.3 


where  c(f^  ^)  is  the  complexity  of  one  evaluation  of  ^ 

e and  c(f^  ^ » 1,  (12.8)  becomes 


(12.9)  comp(Y,S,T,e)  — 


For  small 


4e 


1,  k 1‘ 


This  shows  that  asymptotically  in  e,  the  e-complexity  comp(Y,S,T, e)  is  a de- 
creasing function  of  the  regularity  parameter  k. 

We  conclude  this  section  by  the  following  remark.  It  is  well  known 
that  it  is  impossible  to  find  an  e- approximation  to  the  integral  of  f knowing 
only  the  values  of  f and/or  its  derivatives  at  arbitrary  but  finite  number 


of  points.  The  argument  is  that  adding,  for  instance,  the  polynomial 
m 2 . 

.u(t)  = c | | (t-t.)  J to  a function  f,  one  does  not  change  the  information 

i=l  1 -1 

on  f,  21(f)  = 2l(f+oi)  , but  the  value  of  integral  J (f  (t)  + ix)(t))dt  can  be 

arbitrarily  different  from  the  integral  of  f.  In  our  setting  we  rule  out 
this  argument  since  we  consider  functions  whose  kth  normalized  derivative 
is  bounded  by  one.  We  show  that  provided  k 2 1,  then  the  integration  prob- 
lem can  always  be  solved  to  within  any  e and  that  at  worst  the  complexity 
goes  as  l/e. 

Compare  with  the  result  of  Bojanov  [74]  who  shows  that  for  analytic  f 


and 


T - I,  comp(Y,S,T,e)  - CKc^log  -)  ). 


i 


i 


13.1 


> 


13.  EVALUATION  OF  A FUNCTION  AND  BOUNDARY  VALUE  PROBLEMS 

Let  = 3o  = (f : f is  continuous  on  [0,1  ] and  f'  € L^O.l)}  with  the 

uniform  norm  j|  f||=max  |f(x)|.  Let  3,  = L (0,1).  Define 
13x^1  ^ 

1 

(13.1)  (Sf ) (x)  = j*  G(x,t)f(t)dt,  Tf  = f' 

0 

where  G s 0,  G ^ 0 is  a continuous  function  on  [0,11  X "0,1].  The  informa- 
tion operator  21  is  given  by 


(13.2)  21(f)  = [f(t0),f(t1),...,f(tr>fl)]t 

where  t^  = ih , h = l/(n+l)  and  card(2l)  = n+2 . 

Thus  we  want  to  approximate  the  function  (13.1)  knowing  the  value  of 
the  integrand  f at  some  points  and  the  bound  ||  f'  |]  £ 1.  Define 


(13.3)  f(t) 


f (tQ) 

f(Ci) 


(f(t 


n+1 


0 £ t < h/2 , 

t^-h/2  S t < t + h.'2 , i = 1,2 

t -h/2  s t s 1. 
n+1 


.n, 


Thus  f is  a piecewise  constant  function.  Note  that  f(t.)  = f(t.)  and  Tf  = 0 

l i 

almost  anywhere.  Since  f does  not  belong  to  3q  we  cannot  use  Lemma  8.3. 
However  we  can  still  show  that  the  linear  algorithm 


(13.4)  cp(2l(f ) ) = Sf 


h/2 


f ^0^ 


G (x , t) dt  + 


£<W 


1— la/ 2 


G(x,  t)  + 


n 


+ 


4f^i> 

i=l 


t +h,/2 

f C(x, t)dt 
t -h/2 


is  an  optimal  error  algorithm.  Indeed 


13.2 


Sf  - cp(3t(f)) 


n 


i^O 


t.+h  2 

X. 


G(x,C) (f (t)-f (t . ) ) dt 

w i 

t . 


+ 


ci+i 

r G(x,t) (f(t)-f(t.))dt 

t i+h/ 2 1+1 


Since  f(t)  - f(t^)  = f (t , t ) (t- 1 , with  the  first  divided  difference 

j f (t , t ) j £ 1 , we  get 


(13.6)  ||  Sf-cp(!t(f))||  * ||  j G (x,  t)  q ( t)  dt  ||  -||  Sq  | 

0 


where 


(13.7)  q ( t) 


t.  £ t £ t.+h  2 

1 l 


t.  -t  t.+h/ 2 £ t £ t.  _ . 

1+1  1 1+1 


Note  that  q(tj  =•  0 for  i =*  0,l,...,n+l  and  Tq  * +1  almost  everywhere.  Thus 
q £.  her  '31  and  ||  Tq||  ^ 1.  From  Lemma  5 . 1 we  get 


||  Sf-<p(3t(f))||  s]  d(31, S ,T)  £ r (31,S,T) 


which  proves  optimality  of  cp.  It  is  easy  to  verify  that 


(13.8)  e (cp)  - r(Ut,S,T)  - ||  Sq  ||  = cnh 


where  c = c (G)  and  there  exist  two  constants  c = c(G)  > 0 and  c 3 c(G)  < + 
n n — — 

such  that  c £ c £ c for  all  n. 

— n 

We  analyze  the  complexity  of  evaluation  of  (13.1).  Let  'f  be  the  class 
of  all  information  operators  of  the  form  (13.2)  for  some  n.  Let  n ■ n(e) 
he  the  smallest  integer  such  that 
c 


(13.9) 


n 

n+1 


< «. 


00 


Due  to  (13.8)  we  get  c(G)/e  < n(e)  + 1 £ c(G)/e  + 1.  Then  the  e-complexity 
comp(Y,S,T,e)  satisfies 


13.3 


(13.101  (c^l)  (n(e)+2)  - 1 s Comp('f,S  ,T,  e)  s (c  ^2)  (n(  e)+2)  - 1 

where  c^  is  the  complexity  of  one  function  evaluation.  For  small 
e and  c1  >•  1,  (13.  101  becomes 


(13.111  comp(  i ,S,T,el  S:  — . 


It  is  possible  to  generalize  this  result  and  to  show  that  if 


k-  1 


3^  “ '.f : f 6 C (0.1)  and  £ L<o(0,l)}  and  Tf  = ^7-  f ' * there  exists  a 
linear  optimal  error  algorithm  cp  which  uses  the  information  (13.2)  with 
n 2 k.  such  that 


1 *00 


(13.12)  e(-p)  = r(Sl,s,T)  = 0(hk)  , comPC?,S,T,e)  = c( 

* 


Furthermore  it  is  possible  to  show  that  for  any  distribution  of  points  t^ 
in  (13.2)  there  exists  a positive  constant  a,  such  that  r(?l,S,T)  2 a,h  for 
all  n 2 k.  Thus  choosing  the  points  equally  spaced  does  not  change  the 
dependence  on  h. 

We  note  that  the  solution  operator  S defined  by  (13.1)  can  describe  a 
boundary  value  problem.  Indeed,  if  G is  the  Green  function  of  a boundary 
value  problem  then  a(x)  = (Sf) (x)  is  the  solution  of  this  problem.  For 
instance  if 


(13.13)  G(x,t) 


t(l-x) 


x(l-t) 


0 S t s x, 
x ^t  si, 


then  "v  satisfies  the  boundary  value  problem 


(13.14) 


( »"( 

I or(0 


x)  ■ -f(x) 


for  x £ (0 , 1) 


(0)  - *(1)  - 0 . 


* 


14 . 1 


14.  EVALUATION  OF  DERIVATIVES 

Let  = ^(-1,1),  ^ = Ck  ^(—1,1)  and  C(-l,l)  for  an  integer 

k 2 1 with  the  uniform  norm  ||  f ||  ■ sup  |f(t)|.  Let 

- KKl 

(14. 1>  Sf  ■=*  f'  (0)  , Tf  *»  ~ f(k) 

and  let  the  information  operator  21  be  given  by 

(14.2)  2t(f;h)  = [f(0) ,f (h)  ,f(-h)  ,...,f(nh)  ,f(-nh)  ]C 

where  the  parameter  h £.  (0,l/n),  card(2l)  = 2n4-l. 

Thus,  for  a given  h we  want  to  find  an  approximation  to  f' (0)  knowing 
function  evaluations  at  the  points  jh  for  j =*  0,+l,...,+n.  Note  that 
round-off  error  analysis  indicates  that  h should  not  be  too  small. 

For  the  sake  of  simplicity  we  solve  this  problem  for  k = 2n+l.  Note 
that  ker  91  H T ■ {o}  c ker  S.  Let 


(14.3)  gj(t)  - 7c_^'7jh7  where  ^c)  ■ c ric^-iV) 


for  j - 0,+l,...,+n.  Note  that 


(14.4)  g^(0) 


0 *.(Q)  „ (HIT  , „+1 

°*  V jh(tH-j) '.  (n-  j) J ’ 


,+n. 


Since  g^(ih)  ■ 6^  and  Tg^  ■ 0,  Lemma  8.3  assures  us  that 

n « n , _ 

2 •<  ^ 

(14.5)  *p<31(f ;h)>  - 2_£Oh)Sgj<t)  -iiV-  £_  <f()h)-£(-]h)) 

j— n j“l 


is  a linear  interpolatory  optimal  error  algorithm.  The  algorithm  (14.5)  is 


I 


14  .2 


known  as  the  nth  central  difference  formula.  We  seek  e(rp).  Since 
n 

f(t)  - f(jh)g.(t)  = g(t)'u(t)  where  g(t)  is  the  kth  normalized  divided 

j=-n 

difference  of  f and  jg(t)|  ^ 1,  then 

f (0)  - cp(2l(f  ;h) ) = g(0)'Jo'  (0) . 

This  yields 

(14.6)  e(-i)  = r(2t,S,T)  = (n'.)2h2n. 

Werschulz  [77b]  considers  the  dependence  of  e(:p)  on  h and  says  <p  has 
order  of  accuracy  p if  e(^>)  = 0(hp)  . The  equation  (14.6)  agrees  with  his 
result  that  every  algorithm  which  uses  the  information  (14.2)  has  order  of 
accuracy  no  greater  than  2n. 

We  analyze  the  complexity  of  evaluation  of  derivatives.  To  find  an 
e-aDproximation  to  f' (0)  for  every  f G,  we  require 

(14.7)  e(cp)  = (n'.)2h2n  < e. 

With  h,  e fixed  this  determines  n.  Note  that  h might  be  chosen  as  small  as 
possible  consistent  with  good  round-off.  If  (14.7)  holds  then 

(14.3)  (2n+l)  (c  x+l)  - 1 s comp(’Jl,S,T,e)  £ (2n+l)(c1+2)  - 1 

where  c ^ is  the  complexity  of  one  function  evaluation. 


15.1 


15.  NONLINEAR  EQUATIONS 

There  exist  problems  for  which  we  usually  do  not  use  f ixed  information 
?l.  For  instance,  the  solution  of  a nonlinear  equation  f(x)  = 0 is  often 
approximated  by  an  iterative  algor  itlm  which  repetitively  uses  an  information 
operator  21(f;x^)  where  x^,  i = 1,2,...,  is  a current  approximation  to  the 
sought  solution.  Information  operators  of  the  form  2l(f,x J are  studied  in 
Traub  and  Wozniakowski  [77c].  Here  we  want  to  show  that  it  is  also  possible 
to  deal  with  nonlinear  equations  for  a fixed  information  operator.  We  shall 
show  that  some  known  iterations  are  "asymptotically"  optimal  error  algorithms. 
Since  this  problem  is  nonlinear  we  cannot  use  the  results  of  Sections  4 
through  9.  Our  analysis  will  be  based  mostly  on  Section  2. 

Let 

(15.1)  f:  D C bx  - B2 

where  D * [x:  ||  x||  < 2R}  and  B^,  are  Banach  spaces  over  the  real  or  com- 
plex fields  of  dimension  m,  m = dim(Bp  =*  dim(B9)  , 1 £ m ^ + ®.  Let  ^ be 
the  class  of  all  operators  f which  are  k- times  differentiable  in  the  Frechet 
sense  on  D,  k ^2.  Define 

(15.2)  3q  ■ 3q(A2  ’A^)  “ [f:  f €.  and  there  exists  a - ar(f)J||  a||  £ R,  such 

that  f(a)  = 0 and  ||f'  (a)1  — 4— ^[|  < A and 

(k)  1 1 

||  f’(a)'1  fk-7  ||  < \ for  all  x€  d} 

for  constants  and  A^  which  satisfy  the  condition 

(15.3)  2kAk(3R)k_1 


+ 2A2R  < 1. 


15.2 


The  solution  of  nonlinear  equations  in  is  described  by 

(15.4)  S(f)  = f_1(0),  X = B,. 

We  first  show  that  S is  well-defined.  It  suffices  to  prove  that 
f(x)  = 0 for  ||  x ||  £ R has  a unique  solution  y = f * (0)  , for  f £ 3g  • Let 

(15.5)  R (x,y;f)  =>  J f^(y  + t (x-y) ) (x-y)  ^ — dt 

J o 

for  x,y  £ D and  j £ k.  Then 


(15.6)  f(x)  = f(a)  + f'(a)(x-o)  + R2(x,o;f) 


and  f(x)  = 0 is  equivalent  to  the  equation 


x-y  = -f'(a)  lR2(x,y;f) , 


From  (15.2)  and  (15.3)  we  get  for  j|  x||  ^ R, 


x-of||  * A j|  x-Or|f  < A_R ||  x-»||  =£^| 


X--V 


which  implies  x = a.  Thus  S is  well-defined  by  (15.4)  and  y = S(f)  satisfies 
the  nonlinear  equation  f(a)  = 0. 

Define  the  information  operator 

(15.7)  ^(f)  = [y(f),f(y(f)),...,f(k"1)(y(f))]t 


where  y = y(f)  is  an  approximation  to  the  solution  a = S(f),  ]|  y|j  £ R. 

We  want  to  find  d(2t,S),  the  diameter  of  information  Ul  for  the  problc 


S . See  (2.9)  . 
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Lemma  15.1 


ZAV  5 If 

(15.8)  d(2t,s>  i — SUP  (~\\  y(f)-S(f)  |n. 

A2K 


?roof 


Mote  that  21(f)  =■  21(f)  implies  f (y)  = f^J  (y)  for  j = 0,1,..., k- 1 and 


f,  f €.  3g*  Then 


f(x)-f(x)  = R^(x,y,f-f) 


Since  f(x)  = 0 is  equivalent  to  f (x)  = R,  (x,y,f-f)  and  f satisfies  (15.6), 


we  get 


(15.9)  x = H(x)  =—  m + f'  (a)  1 {^(x.y, f-f)  - R (x,ar,f)]. 

We  show  that  H is  a contraction  on  J 3 fx:  j|  x-o||  s j||  y-»||  }•  Indeed, 
||  H(x)-ar||  * 2Ak||  x-y|f  + A£  ||  x-a| f < 2Ak(|)k||  y-crlf  + ^ ||  y-of |P  * 
jll  y~m\\  (6Ak(3R)  -4-  A^R)  £ — J|  y- o 1 1 due  to  (15.2)  and  (15.3)  Furthermore 

||  H’  (x)  ||  * 2kAk  ||  x-ylf'1  + 2A2||  x-a||  * 2kAk(2R)k_1  + 2A2  R < 1 
due  to  (15.3).  Thus  the  equation  (15.9)  has  a uni_que  solution  5, 

IPH  5 |ll  y-»||  * R-  Set  x = 5 in  (15.9).  Then  ||  ^»||  s 2AJI  S-y  |f  + A2 


which  yields 


or- or  ^ 


2Ak(-||  y-*|D 

1 - A2R 


This  proves  (15.8)  and  completes  the  proof. 


want  to  prove  that  (15.8)  is,  in  general,  sharp  with  respect  to  |^(f)-S(f) if. 


Lemma  15.2 


If  y(f)  approaches  a « S(f)  then 
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(15.10)  diam(21,S)  = 2A^  sup  [ (|  y(f)-S(f)  |f(l  + o(||  y(f)-S(f)  ||)}. 


f£3, 


Proof 


The  equation  (15.9)  for  x - a yields 

(15.11)  ||  5- a ||  ^ 2AjJ|  y(f)-a|fC  + A2||  5-a|f  . 

Since  ||  crcif||  = 0(  ||  y(f)-a|fC)  , (15.11)  can  be  rewritten  as  ||  5-  ar| | s 
2aJ|  y(f)-rv|fC  (1  + o(l)).  Since  this  bound  is  sharp  we've  proven  (15.10).  ■ 
Lemma  10.2  states  that  the  diameter  diam(9l,S)  is  roughly  equal  to 
2 A,  sup  ||  y(f)-S(f)  if1  where  k is  the  first  omitted  derivative  in  the  informa- 
tion  (15.7)  . 

We  establish  asymptotically  optimal  error  algorithms  for  the  problem  S. 

Let 

(15.12)  f(x)  = f (y)  + f'(y)(x-y)  + ...  + (kTi)''.'  ^(yMx-y)1* 

Note  that  51(f)  = 51(f).  From  (3.10)  in  Traub  and  Woz'niakowski  [77b]  we  know 
that 


f_1(3) 


1 f"(x) 

kl 


b + 


y-o/||  ) 


k-2 


y-'vlhkAk(f  II  y- ^ 1 1> k 1 


* a2  (y) . 


Thus  f & (y)  »0)  where  A2(y)  = A2  + 0(||  y-»||  ).  Define  the  algorithm 

(15.13)  cp(51(f))  = S(f), 

i.e.,  '*(5l(f))  Is  a unique  solution  of  the  nonlinear  equation  f(x)  = 0.  The 
, . rithm  115.13)  is  known  as  the  interpolatory  iteration  I and  was  con- 
i.  Iraub  and  Woz'niakowski  [7&>.  77a,  77b].  Note  that  for  n = 2 we 


J 
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get  one  step  of  Newton  iteration  since  f (x)  = f(y)  + f'(y)(x-y)  and 
9(2t(f) ) = 5 = f(y)  - f ' (y)  1f(y). 

Lemma  15.3 

(i)  For  any  y(f)  the  error  of  cp  is  bounded  by 
A 

(15.14)  e((p)  St—  sup  (§  ||  y(f) -S  (f)  ||  ) k. 

i-A2K  f£30 

(ii)  If  y(f)  approaches  a = S(f)  then  the  algorithm  9 is  asymptotically 
optimal,  i.e. , 

(15.15)  e(ro)  ^ r(2t,S)  — A,  sup  |j  y(f)-S(f)  if".  ■ 

f£30 

Proof 

To  prove  (15.14)  we  repeat  the  proof  of  Lemma  10.1.  Note  that  R^(x,y,f-f) 

in  (15.9)  for  f defined  by  (15.12)  has  the  bound  ||  R^(x,y,f-f)  ||  5 A^||  x-y|f 

which  yields  (15.14).  The  same  argument  enables  us  to  conclude  tnat 

e (9)  sup  ||  y(f)-S(f)  if  for  y(f)  approaching  a.  Thus,  from  Lemma  15.2 

f *0 

we  get 

e(9)  ^ --  d(!R,S)  r(5l,S) 

which  proves  (15. 15) . | 

The  algorithm  (15.13)  is  known  to  have  maximal  order  of  convergence 
among  all  iterations  using  the  information  of  (15.7);  see  Traub  and  Wozniakowski 
[76a].  Lemma  15.3  states  that  this  algorithm  has  asymptotically  optimal  error 
in  the  class  3g- 

Complexity  of  the  algorithm  (15.13)  and  its  dependence  on  k were  con- 
sidered in  detail  by  Traub  and  Wozniakowski  [77b]. 
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CHAPTER  IV 
CONCLUDING  REMARKS 

16.  COMMENTS  AND  EXTENSIONS 

This  is  the  first  of  a series  of  papers  in  which  we  develop  an  informa- 
tion based  theory  of  optimal  error  algorithms  and  of  problem  complexity. 

We  conclude  this  paper  by  a partial  list  of  interesting  problems  and  exten- 
sions which  will  be  studied  in  the  future. 

1.  In  this  paper  we  restrict  ourselves  to  general  information 
operators  St  = 21(f)  and  illustrate  our  concepts  and  results  by 
simple  examples.  Future  papers  will  be  devoted  to  complete  analy- 
sis of  an  application  area  in  this  general  framework.  We  will 
include  the  effect  of  computing  21(f)  approximately,  problem  con- 
dition, algorithm  stability,  and  the  cost  of  arithmetic  precision. 

2.  For  some  problems  information  operators  have  additional  structure. 

For  instance,  "iterative"  information  operators  21  = 3l(f,x.,x.  x ) 

li-l  i-m 

depend  on  current  approximations  to  the  solution  and  they  are  used 

for  i = m,nH-l,...  . The  parameter  m measures  the  size  of  "memory" 

used  in  the  process.  For  m - 0 we  have  information  operators 

"without  memory",  for  m ^ 1 "with  memory".  See  Traub  and  Wozniakowski  [77q1. 

3.  In  Chapter  2 we  deal  with  linear  information  mostly  of  the  form 

31(f)  =■  [L^  (f)  (f) , . . . ,Ln(f)  ]*"  defined  by  n independently  given 

linear  functionals  Lj,  jLg, . . . ,L  . A natural  generalization  is  an 
"adaptive"  linear  information  operator  of  the  form 


16.2 


«(f)  = [Lx(f) ,L2(f;L1(f)),...,Ln(f;L1(f),...,Ln_1(f))]t 


where  depends  linearly  on  its  first  argument.  This  form 
enables  us  to  use  the  previously  computed  functionals  to  deter- 
mine the  next  functional.  Such  adaptive  information  is  widely 
used  in  practice  in  a number  of  application  areas. 


4.  In  Chapter  2 we  deal  with  linear  problems  defined  by  two  linear 

operators  S and  T.  The  restriction  operator  T defines  the  domain 

1 k 

of  problem  elements.  In  several  applications  we  take  T = 7-7  D , 

K • 

the  kth  normalized  derivative  operator,  and  we  show  that  complexity 
decreases  as  the  regularity  parameter  k increases.  We  believe  this 
| to  be  a general  phenomenon. 

5.  For  some  problems  the  domain  3q  i-s  defined  by  a two  sided  inequality 
on  T,  i.e.,  3g  = [f  €.  3^:  0 < c ^ ||  Tf||  £ l]  for  a constant  c. 
Sometimes  3q  is  defined  by  more  than  one  linear  operator,  e.g., 

3q  = {f  c.l  s II  Tj_f||  s 1»  f°r  i = 1,2, ...,m)  for  nonnegative 

constants  c,,c  c . 

1 Z m 

6.  All  basic  results  of  Chapter  2 are  for  a linear  operator  S.  Much 
of  the  linear  theory  can  be  generalized  to  nonlinear  operators  S. 

For  instance,  assume  that  there  exist  two  linear  operators  and 

such  that 

II  s1(f1-f2)  II  £ II  S(f1)-S(f2>  II  5 ||  s2(frf2)  II 

for  all  f^  and  f2  from  the  domain  of  S.  Then 
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d^.sp  ^ d(Sl,S)  * d(«,S2) 

and  one  can  apply  the  linear  theory  to  and  S2  to  derive  lower 
and  upper  bounds  on  the  diameter  d(2l,S)  and  complexity  of  the 
problem  S. 
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GLOSSARY 

We  summarize  below  basic  concepts  used  throughout  the  paper.  We  list 
a symbol,  its  meaning,  and  the  section  reference  where  this  symbol  appears 
for  the  first  time. 


Symbol 

Mean ing 

Section  Reference 

S 

the  solution  operator,  sometimes  called  the 
problem,  S:  ^ ^ and  3Q  c:  ^ 

2,  (2.1) 

^0 

the  domain  of  S 

2,  (2.1) 

linear  space,  3q  c 3^ 

2 

^2 

the  range  of  S 

2 

e 

error  parameter,  e > 0 

2 

x = x (f ) 

e-approximation,  ||  x-a||  < e. 

2,  (2.2) 

f 

the  problem  element,  f €.  3q 

2 

a 

the  solution  element  a = S(f) 

2 

31 

the  information  operator,  21;  Dyj  -»  ^ 

2,  (2.3) 

the  range  of  31 

2,  (2.3) 

d(3t,S) 

the  diameter  of  information  31  for  the  problem  S 

2,  (2.9) 

r(3t,S) 

the  radius  of  information  31  for  the  problem  S 

2,  (2.10) 

CP 

algorithm,  cp:  31(3^  - 32 

2 

e(cp) 

the  error  of  algorithm  cp 

2,  (2.13) 

the  class  of  all  algorithms  using  the  informa- 
tion 31  for  the  problem  S 

2 

I 

9 

interpolatory  algorithm 

2,  (2.16) 

e(2l,S) 

the  optimal  error 

2,  (2.18) 

oe 

9 

optimal  error  algorithm 

2,  (2.19) 

c 

9 

central  algorithm 

2,  (2.23) 

» 
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Symbol  | 

bection  Reference 

P 

the  set  of  primitives 

3 

comp  (21(f) ) 

the  information  complexity  of  computing  21(f) 
where  21  is  a permissible  information  operator 

3 

comp  (cp(y)  ) 

the  combinatory  complexity  of  computing  y 
where  cp  is  a permissible  algorithm 

3 

$(e) 

the  class  of  all  permissible  algorithms  for 
which  e(cp)  < c 

3 

r(2l,S)  2 e 

the  problem  S with  information  21  is  e-non- 
compu table 

3 

r(2t,S)  < c 

the  problem  S with  permissible  21  and 
®(e)  / f)  is  e-computable  with  respect  to  P 

3 

comp  (cp) 

the  complexity  of  an  algorithm  cp 

3, 

(3.1) 

comp(21,s  ,e) 

the  e-complexi/t  of  the  information  21  for  the 
problem  S 

3, 

(3.2) 

oc 

9 

optimal  complex-  algorithm 

3, 

(3.3) 

comp  (21) 

the  information  complexity 

3, 

(3.4) 

Y 

a class  of  permissible  information  operators 

3, 

(3.8) 

comp(Y,S,e) 

the  e-complexity  of  the  problem  S in  the  class 

Y 

3, 

(3.9) 

“l  C 'Jl2 

ker  21^  c ker  21^ 

4, 

def.  4.1 

ker  21  = ker  21 

1 2 

4, 

def.  4.1 

AX 

algebraic  complement  of  A 

4, 

(4.2) 

codim  A 

codimension  of  A 

4, 

(4.2) 

card  (21) 

the  cardinality  of  the  information  21 

4, 

(4.5) 

T 

the  restriction  operator,  T:  3^  ”* 

5, 

(5.1) 

the  range  of  T 

5, 

(5.1) 

d(21,S,T) 

the  diameter  of  information  21  for  the  problem 
(S.T) 

5 

index (S,T) 

1 

the  index  of  the  problem  (S.T) 

5, 

def.  5.1 

I 
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Symbol 

Meaning 

Section  Reference 

A(S,T) 

algebraic  complement  of  ker  T fl  ker  S in  the 
space  ker  T 

5, 

(5.3) 

* * 
§!»••• >Sn* 

★ 

basis  of  A(T,S),  n = index(S,T) 

5, 

(5.3) 

★ 

31 

information  operator  such  that  card (31  ) « 
index (S,T)  and  ker  31*  f)  ker  T c ker  S 

! 6> 

(6.1) 

Y 

n 

the  class  of  all  information  operators  31  such 
that  3l*c  51  and  card  (31)  s n 

6 

T'1 

the  inverse  operator  of  T 

6, 

(6.3) 

d(n,S,T) 

the  n-th  minimal  diameter  of  information 

6, 

(6.6) 

31 oi 
n 

n-th  optimal  information 

6, 

(6.7) 

K 

the  linear  operator  K =*  ST  * 

6, 

(6.9) 

b(m,K) 

the  m-th  minimal  norm  of  the  linear  operator  K 

6, 

(6.9) 

B 

m 

m-th  minimal  subspace  of  the  linear  operator  K 

6, 

(6.11) 

d(S,T) 

the  diameter  of  problem  error 

6, 

def.  6.2 

d(S,T)  =>  + “ 

the  problem  (S,T)  is  strongly  non- computable 

6, 

def.  6.2 

d(S,T)  2 2« 

the  problem  (S,T)  is  e-non-computable 

6, 

def.  6.2 

d(S,T)  =■  0 

the  problem  is  convergent 

6, 

def.  6.2 

a(Y,S,Ttc) 

the  «-cardinality  number  for  the  problem  (S,T) 
in  the  class  Y 

9, 

(9.5) 

Y 

U 

the  class  of  all  linear  information  operators 

31  such  that  card  (31)  < + 00 

9 
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theory  rationalizes  the  synthesis  of  algorithms  by  showing  how  to  construct  al- 
gorithms which  minimize  or  nearly  minimize  the  error.  For  certain  classes  of 
problems  it  shows  how  to  construct  algorithms  (linear  optimal  error  algorithms)'] 
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20.  abstract  (continued) 

which  enjoy  essentially  optimal  complexity  with  respect  to  all  Possible  al- 
gorithms. The  existence  of  "strongly  non-computable"  problems  is  demonstrated. 
In  contrast  with  the  gap  theorem  of  recursively  computable  functions  we  show 
that  "every  monotonic"  real  function  is  the  complexity  of  some  problem. 
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